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ABSTRACT 

This report documents the theory, structure, and 
implementation of a performance processor (written, in FORTRAN IV) 
that can accept performance demonstration data representing various 
levels of operator's skill and, under user control^ analyze data to 
provide candidate performance measures and validation test results. 
The processor accepts two types of information: (1) sample 
performance data on magnetic tape, and (2) user information 
reflecting knowledge about features of the performance that are 
considered to be important to measurement. The sample performance 
data input is smoothed by the processor in order to remove or reduce 
noise factors in accordance with information provided by the user. 
Criterion performance functions are^ optionally, provided by the user 
or are computed by the processor using skilled performers' data. The 
processor then develops a discrete representation of the continuous 
performance data based on observed deviations, from the criterion 
functions. This discrete representation, in turn, is used to model 
each performance using state-space techniques. The processor operates 
on the state-space model to compute vectors which form generators of 
various conceivable measure spaces. Candidate performance measures 
are then generated by operating on the vectors with multiple 
regression algorithms. Empirical validation tests of several types 
are applied to the candidate measures for assessment of their 
validity- likelihood. (Author/DEP) 
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SUMMARY 



Problem 

The problem was to develop and implement standardized techniques for deriving and validating 
measures of operator performance. Traditional techniques involve hand-selecting measures which appear to 
have content validity, then testing the measures against other validation criteria using operator performance 
data. This usually results in a resource-consuming iterative research process that is often unsuccessful, 
because: (1) it is never known at the onset whether or not the most useful measures have been overlooked, 
(2) the number and potential validity of measures investigated are limited by and vary with the researcher's 
ingenuity and the time he has available for the study, and (3) the research process and all associated manual 
effort must be repeated for each new measurement task. 

Approach 

The approach was to develop and implement computer-aided techniques for deriving and validating 
operator performance measures. A "universal'* set of potential measures .was defined which possesses 
characteristics encompassing many traditionally selected measures. The set also inherently contains a 
myriad of other measures whose characteristics render them reasonable candidates. Vectors were then 
identified which constitute generators for the set of measures (i.e., the vectors span the defined measure 
space). Computational algorithms were developed which generate and operate on the constituent vectors 
using multiple regression techniques. Several empirical validation methods were developed for testing 
candidate measures thereby generated. All techniques were implemented in a computer-aided measurement 
processor which: (I) accepts sample performance data and various user inputs, and (2) generates and tests 
candidate measures, computes* statistics for assessing their validity likelihood, and prints results for user 
analysis. ♦ 

Results 

The developed measurement processor was successfully implemented on a Sigma 5 computer. 
Demonstrations of the operation of the software were performed using a limited amount of pilot 
performance data recorded on a T-37B aircraft. Tlie processor performed necessary data smoothing, 
automatically segmented the Hight maneuvers for measurement, and developed criterion functions, from the 
skilled operator data provided. Actual generation and validation of measures was not demonstrable due to 
nonavailability of originally anticipated data. However, correct software performance of all parts of the 
processor was verified. 

Conclusions 

Tlie theoretical concepts and computational techniques underlying the developed measurement 
processor are unique and have great potential for operato'' performance measurement research. The applied 
concept of developing a set of vectors which span a conceived measure space and operating on it with 
regression techniques to generate candidate measures is itself suggestive of a new and extremely powerful 
measurement tool. The processor operation can be largely independent of user intervention; however, it is 
also capable of accepting user inputs refiecting his knowledge about specific measurement problems. It 
represents a truly interactive research system wherein user tasks as distinguished from processor tasks are 
logically defined, and the outcomes of each are integrated. 

fivaluation of the adequacy of the spanned measure set, the generating vectors, and the computational 
mechanisms for generating and testing measures could not be performed as originally planned due to 
nontechnical problems which prevented the collection of required data. This was extremely detrimental to 
the study because: (1) many of the techniques could not even receive preliminary test prior to their 
incorporation in the processor, and (2) the contributions made by this study to the general technology can 
only be suggested instead of exemplified. 

Follow-up research should include derivation of the basis of the defined measure set using the 
implemented processor as an aid to empirical studies. This is, in essence, the real crux of the operator 
performance measurement problem. 

G 
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COMPUFER-AIDED TECHNIQUES FOR PROVIDING 
OPERATOR PERFORMANCE MEASURES 



I. INTRODUCTION 



This report describes underlying theoretical concepts and computer-implemented techniques for 
deriving valid and objective operator performance measures. Tlie original impetus for the work came from 
requirements for measures of pilot performance; however the techniques and development-concepts are 
equally applicable to general assessment of operator p**vforniance on continuous control tasks. Therefore, 
the basic mathematical and computer techniques will be described in a general coniext, while (limited) 
example data are presented for selected.pilot performance tasks, 

A Tale to Illustrate Basic Concepts 

Suppose we are faced with the relatively simple task of deriving and validating measures of 
performance on the terminal portions of a ground controlled approach (GCA), Let us further simplify the 
problem by restricting it to measurement of the pilot's ability to maintain proper altitude during descent to 
the runway. Following a typically employed course of action, we might begin by specifying an intuitive 
(accepted) notion of "ideal" performance and perhaps sketch some hypothetical performance profiles, such 
as those shown in Figure 1. 

The next step would typically be to identify candidate measures which, singly or in combination, and 
in whole or in part, we expect will solve the problem. Thus, we might logically pick: (1) RMS Glideslope 
deviation, (2) Maximum glideslope deviation, and (3) Time in tolerance (glideslope ± A) as measures to 
compute and examine for validity. 

We would then probably perform some initial data collection, and study the behavior of these 
selected measures for various pilots (perhaps some novice and some at various other stages in the range from 
novice to experienced). We might discover (again assuming a typical case) that one of the selected measures 
tends to discriminate between some of the novice and highly experienced performers, but not in all cases; 
and that none of the measures say anything conclusive or consistent about performers whose experience 
level (and/or subjectively judged skill level) lies between the two extremes, 

"Aha!'' says our colleague, "The reason your RMS doesn't work well is because glideslope deviations 
close to the ground are more critical than deviations at higher altitudes. You need to take altitude into 
consideration and weight the deviations accordingly," 

"And I know why maximum deviation didn't work out," says another, "The deviation doesn't matter 
as much if it is above the glideslope as if it is below. You should take deviation direction into account," 

"Your time in tolerance looks like it might be OK if you would just change the tolerance value to be 
more in line with the way our good pilots actually perform. And maybe the tolerance should be 
variable — perhaps a function of altitude — because tighter control is critical as you near the threshold." 

Well, no wonder things looked so bad on the initial study! As a result of this first iteration, we might 
be well advised to plot some of the actual performances and reconsider the problem altogether. We could 
discover, for instance, that one thing unique about the 5 least experienced pilots for whom we have some 
data is that they oscillate about the glideslope considerably more than the skilled performers do, (Maybe 
the number of glideslope crossings would be a good measure!) We might also observe that the more 
experienced pilots, when they do deviate significantly from the glideslope, make very gradual corrections, 
whereas the novice performers tend to correct more rapidly, and they often overshoot, (Maybe rate of error 
correction would work!) Finally, we may see that the good pilots (except for 3) never descend below the 
glideslope, even though deviations above it are sometimes rather large, (Maybe whether or not descents 
below the glideslope occur at all would provide at least part of the answer , , , , or might this just be a 
characteristic of a cautious pilot?) 

At this point, our original list of 3 potential measures has tripled. We have now identified the 
following 9 measures for investigation: 

1 . RMS glideslope deviation — unweighted 
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2, RMS glideslope deviation - weighted by altitude 

3, Maximum glideslope deviation - unweighted 

4. Maximum glideslope deviation - weighted by direction of deviation 

5, Time in tolerance (Tol. = ± A) 

6. Time in tolerance (ToL = f (altitude)) 

7, No. of glideslope crossings 

8. Rate of error correction 

9. Whether or not descents below the glideslope occur 

Iterations 2 through K of the study would be similar in nature-to that described above, and typically, 
the number of candidate measures would vary and grow multiplicatively in direct proportion to the number 
of iterations we are able or willing to conduct. The concluding actions of the study, agam if typical, would 
be one or more of the following: 

(a) Documentation of the work performed and a recommendation that further study be conducted. 

(b) Determined selection and use of a few of the best-looking measures, with reluctant acceptance of 
the fact that they lack sensitivity and reUability (but by goUy they have to work because they have content 
validity!) 

(c) Reconsider whether we really need neif ormance measurement techniques at all. 

(d) Use some other method of assessing performance, perhaps one that seemed to work OK in the 
1940's or '50's (althougli if it really did work, there would have been no need for this ground controlled 
approach (GC A) study in the first place !). 

The purpose of the preceding tale was twofold. First (although certainly not enliglitening to the 
readers experienced in this area), it iUustrates on a comprehensible scale some of the complexities and 
problems inherent in measurement work, at least as it is commonly approached. To put it simply, the 
researcher is faced with assessing the performance of the most incredibly complex ''black box" conceivable, 
and many times without even the benefit of knowing the standards that should be expected of it as 
distinguished from those that appear, in practice, to be expected of it. (Certainly, much progress has been 
made in rendering the human black box white; however considerably more is required before measurement 
of human performance on real-world tasks can be considered straightforward.) 

The second purpose was to lay the groundwork for a description of the concepts underiying the work 
reported herein. 

Why attempt to identify and laboriously invesrigate a few hand-selected candidate measures, 
repearing the process for each new measurement problem, and never knowing whether or not the measures 
most suitable have been simply overlooked or unconceived? Why not, instead, define a "universal' 
measure-set which encompasses at least the characterisrics represented by the so-called classical measures 
(and then some), and assign to computers the task which are logically theirs; i.e.*, information search and 
retrieval? In other terms, what is suggested is that a measure-set be designed which is, in effect, inclusive of 
measures we typically select for invesrigarion, and, moreover, contains the power to generate a myriad ot 
other potential measures which have either not yet been conceived and/or are too numerous to list for 
purposes of hand-selecring those that seem appealing. This is feasible, and the measures in such a set are 
reasonable to investigate if the characterisrics of the set are defined rationally. 



II. APPROACH 



The approach is to develop a trial measure-set encompassing characterisrics common to many of the 
classical measures; and to develop a computer program which generates candidate measures from the set, 
executes various empirical validation tests, and prints results for analysis. 
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Measure Set Summary 

The devised measure set is partitioned into tliree subsets, 'each of which represents measures with 
different characteristics. One subset generates candidate measures which assess performance as 
characterized by unique patterns of performance variables and their frequency characteristics ("Absolute" 
measures). The second generates measures that assess performance as characterized by simultaneous (or 
non-simultaneous) occurrence of unique events ("Relative" measures). The third generates measures that 
assess performance as characterized by unique successions of events or system states ("State Transfer" 
measures) and deviations from standards (state frequency measures), where the standards are either defined 
by the user or computed from user-provided performance data. 



Introductory Example 

In way of example, each of the above mentioned types of measures will be illustrated, using where 
possible the previous example of a GCA approach. An attempt will be made to demonstrate that the three 
types of measures comprising the defined set not only encompass the specific measures of the previous 
example, but conceivably most other measures commonly (or uncommonly) selected for pursuit in 
measurement efforts as well as a host of previously untried ones. 

First it is necessary to mention (with details presented later in the report) that the measures 
computed are based on a discrete representation of the performance data, derived through a transformation 
process. The transformation results in a representation of the data in terms of the number of units by which 
the value of each variable (e.g., roll, pitch, altitude) is displaced from some reference level or reference 
function. (The size of the unit-displacements is determined partly as a function of performance range and 
variance.) Thus altitude, in the GCA example, may be represented by several Boolean functions, each of 
which denotes whether or not altitude lies in a specific band around the glideslope (e.g.,' a band 30' wide 
located 100 above (or below) the glideslope may be represented by one Boolean function). 

States of the (pilot/vehicle) system are represented by the collective states of the various Boolean 
functions over time and, in turn, are represented simply by numbers. Thus the number 6 (binary 1 10), 
depending on the Boolean functions breing investigated, may tell us that at that sampling instant, the pilots 
altitude was 100 ±'15' above the glideslope (first binary digit (1)), his airspeed was 120 knots ± 3 knots 
(second binary digit (1)), and his roll angle was not equal to zero ± 2° (3rd binary digit (0)). 

It is this state representation which allows us to efficiently generate and test measures of Uie 3 types 
described. Any measures of deviation from the reference function (including time in tolerance, for instance) 
are inherent in the collective frequencies with which the various defined states are acquired in performance 
of the maneuver. Any measures of error correction or its rate are inherent in the transfers that occur 
between various states over time. Measures of frequency content of the data (including the number of 
glideslope crossings in the previous example) are inherent in the state transfers that occur and/or in the 
"absolute" type of measure that is investigated. Finally, measures which relate various key events (e.g.,' 
smaller glideslope deviations at lower altitudes) are inherent in the "relative" type of measure that is 
explored. 

Consider, first, the measures of RMS glideslope deviation in the previous GCA example. 
Mathematically, tills is represented as 




RMS = 



where X is the actual altitude and Y the glideslope altitude. Equivalently this relationship may be 

I ^ 

correlatively represented by mean sa^uare = RMS^ ( Y, ^P'^)' 

" i=I ' ' 

where Dj are specific deviations from the glideslope and T are die frequencies with which the 
associated deviations are encountered. Similarly, a weighted mean square error would be 

RMS^ = ~ ( W. Fi Dj^); where W. are weights assigned to each deviation. In the measure-subset based 
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on state frequencies and state transitions, the D. are represented by the number of units displacement from 
the glideslope associated with the Boolean functions representing altitude. Tlie F. values are the computed 
state frequencies depicting the number of observed samples in which the respective Boolean function is true 
(1). 

Since the referenced -measure-subset encompasses F., D., and related vectors, it may be viewed as a 
vector set which spans the space of measures represented by RMS glideslope deviations, weighted or 
unweighted, in addition to any other conceivable measures attending to deviations from a reference 
function (i.e., average deviation, maximum deviation, integrated error, etc). Therefore, the referenced 
measure-subset contains a basis for the space of measures of this type, and any measure of the type 
mentioned may be approximated by some linear combination of the vectors of the defined subset. It is 
maintained that since this is true, all measures of the type spanned by this subset may be explored by a 
computational mechanism wliich can generate the defined vectors and perform multiple regression analyses. 

Next, consider the time in tolerance measures and the measure of whether or not descents below the 
glideslope occur. The former is represented simply by the frequency of occurrence or one or more states, 
fomied by Boolean functions involving the desired tolerance value, A variable tolerance value is 
automatically included in the analysis because the different Boolean functions themselves (and associated 
states) represent different tolerances. The latter is also represented by state frequencies, in that states 
representing descents below the glideslope would all be zero if no descents below the glideslope occurred, 
and non-zero otherwise. Therefore, these two types of measures are included in the computational 
mechanism that explores the measure-subset based on state frequencies and state transitions. 

Finally, consider the measures (in the GCA example) of number of glideslope crossings and rate of 
error correction. Tlie first is represented by the number of transitions that occur between states 
corresponding to aircraft positions below and above the glideslope. The second is also represented by state 
transitions which; (1) distinguish error growth from decay by the identity of the states between wliich 
transitions are occurring, and (2) assess the rate of growth or decay by the relative frequencies of 
between-state transitions and within-state transitions. Therefore, these types of measures, too, are included 
in the state frequency and state transition measure subset and related computational mechanism. 

This single measure -subset therefore covers, all of the specific measures "selected" in the previous 
GCA example and much, more -it covers the general types of measures that are suggested by any 
considerations of deviation from a reference function, steady-state or transitive positions with respect to it, 
ard movement or rate of movement toward or away from it. The potential power, flexibility, and utility of 
a computational mechanism exploring this variety of measures is significant. A recent unique application bf 
the state transition concept in measurement and analysis of performance is described in Connelly and 
Lx)ental(1974). 

We have yet to discuss the other two measure-subsets ("Absolute'' and "Relative''), The "Absolute" 
subset and its respective computational mechanism assesses performance characteristics lelated to the 
repetatlve frequencies, periodicities, and associated patterns of changes between various states. This is 
accomplished in an overall manner similar to that described previously; i.e., vectors which span these types 
of measures are generated and various measures are explored using regression analyses. Examples of 
measures that would be included here are the extent and type of "control diddle" used by an operator; 
frequency characteristics of an operator's ballistic response to, say, a step input; and measures related to the 
number of control reversals used in performing a segment of some task. 

The "relative" subset and computational mechanism fills an identifiable void in the system as thus far 
described. It takes into consideration the proximity in time with which various events take place and the 
conditional probabilities of certain events occurring, given that others have occurred. Again, the approach is 
to generate vectors which span these types of measures and employ regression analysis. Examples of 
measures thereby addressed are whether or not a pilot achieves and maintains straight and level flight 
whenever he is within a specified distance from the threshold on a GCA approach; whether or not he begins 

1.5 
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to roll at the same time (not before or after) he acquires maximum pitch in an aerobatic maneuver; and 
whether or not he characteristically achieves a specific (criterion) airspeed at key points in performance of, 
say, a lazy 8 maneuver. 

Summary 

The preceding subsections describe the fundamental concepts of the approach. The intent is to define 
a "universar* set of measures, not by enumerating all measures in the set, but by defining their 
characteristics. Three major characteristics have been defined, and the evolved measure set is represented 
accfr^'iu^y by three subsets. It has been shown that measures generated from these various subsets 
encoifiiuss a host of typically selected measures (e.g., those of the GCA example) as well as many others 
possessing the subset characteristics. Mathematically, this may be viewed as developing vectors which span 
various measure-spaces, and it is proposed that tlie measures thereby spanned may be explored using 
multiple regression analysis. 



m. BACKGROUND AND STUDY OBJECTIVES 



Many of tfie. basic concepts and mathematical techniques fundamental to this study were explored on 
a trial basis in previous feasibility studies and are documented in the references (Connelly, Schulcr, & 
Knoop, 1969; Connelly, Schuler, Bourne, & Knoop, 1971). However, efficient computer techniques for 
exploring the various types of measure-subsets were never fully developed in previous efforts; and the data 
transformation techniques as well as the measure subsets themselves have been altered and refined for this 
study on the basis of earlier experience with the approach. 

The purpose of the present study was originally to: (1) refine the previously explored technqiues, (2) 
develop efficient computer implementation methods, (3) validate and demonstrate performance of the 
software, and (4) apply the techniques thereby implemented to derive and validate performance measures 
for five training maneuvers flown in T-37B aircraft as part of the Air Force UPT program. Due to 
non-technical difficulties encountered in collecting the required student and instructor-pilot data, part 4 of 
the original objectives had to be abandoned, and the objective substituted in its place was to implement and 
demonstrate the developed software on the Simulation and Training Advanced Research System (STARS). 
(The STARS system is located at the Advanced Systems Division, Air Force Human Resources Laboratory 
(AFSC) Wright-Patterson Air Force Base, Ohio. The associated digital computer is a Xerox Data Systems 
(XDS) Sigma 5.) Therefore, this report documents the computer software developed and the related 
computational algorithms implemented for exploring selected types of measure-subsets; however, since only 
very small amount of data was able to be collected for the study, it was not possible to develop and 
validate any specific measures. The extensive data coUection and reduction machinery developed for use 
(but unfortunately not applied in this study) Is described in Knoop and Welde (1973) and Gregory and 
Cavanagh (1973). 

Scope of Study 

The study includes the development and implementation of 3 different computational mechanisms 
for generating carididate measures from the defined subsets. These are the relative, absolute, and state 
transfer measures previously discussed. A separate computational mechanism for state frequency measures 
was not included, partly because the state transfer mechanism itself generates the state frequency data that 
IS needed. Original plans were to develop and independently test a separate state frequency mechanism 
usmg this generated data and then, depending on results, interface it with the other elements of the 
processor. Due to the previously mentioned change in program objectives and associated lack of 
performance data, however, this was not able to be pursued beyond the planning stage. Emphasis in the 
study, therefore, was on developing and implementing efficient computer techniques for the 3 developed 
computational mechanisms and the overall computer-aided piocessor as described next. 
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IV. COMPUTER-AIDED PROCESSING TECHNIQUES AND SUBSYSTEMS 



The computer-aided automated pilot performance measurement processor is a FORTRAN IV 
program which generates candidate performance measures tlirough various operations on actual 
performance data. These operations include: 

a. Develop performance criteria, 

b. Determine the significance of deviations from criteria, 

c. Transform sample performance data into a compact form for processing, 

d. Conduct a systematic standardized search for candidate perform.ance measures, 

e. Perform validation tests, and 

f. Provide data management processes. 

A generalized flow diagram of the processor appears in Figure 2. 

A primary task in developing performance measures is the determination of standards or reference 
functions. Performance standards should define the unique manner in which the operator should perform 
the task. Often, however, there are a number of satisfactory ways to properly accomplish a task and there 
may exist a family of reference functions representing criterion performance. As a result, the reference 
function forms employed by the processor may accept parameters provided by the user or estimated from 
sample performance data. 

Multi-variable regression is used in formulation of reference functions from sample data. The idea is 
to extract from demonstrations of superior performance functions which uniquely represent that 
performance. Evaluation of the function fit is accomplished through analysis of residues. A small residue 
value indicates a convenient clustering of all superior performance data, while a large residue value indicates 
that the regression formulation is not appropriate or that other parameters are required. 

An additional test of the candidate functions is made by comparing residues obtained from the 
superior performance category data with those obtained from other performance category data such as 
good, fair, and poor. The difference between the residues obtained is an indication of the potential 
performance discrimination capability of a measure developed from that criterion. 

A second important step in the development of performance metrics is the determination of the 
relevance of deviations from the reference performance. It should be noted that the importance of operator 
errors is generally not constant over the entire problem state space. Thus, some systematic means must be 
provided to test various types of deviations and patterns of deviations as to their relevance to performance 
measurement. Table I shows various ways that deviations from the criterion or reference might be related 
to performance measurement. The processor's capability to assess the significance of a wide variety of 
relationships such as these is automatically assured due to the types of performance measures it is designed 
to generate and test. 

The processor has four main portions: 

1. Input and preparation of data, including " 

a. Data management 

b. Smoothing 

c. Maneuver Sectoring 

2. Generation of criterion functions via regression analysis 

3. Processing of data by adaptive mathematical models 

4. Testing and specification of performance measures. 
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Table 1. Some Possible Criterion and Performance Measure Factors 



Type of Criterion 



Possible Ways Deviation (Error) 
is Related to Performance 



Functions Relating Problem 
Variables (Reference Path) 



Differential Reference (where criterion 
is specified by differential or difference 
equations) 



Fixed (variable) tolerance at a specific 
time or at a specific value of another 
variable 



Sequence of Operation 



o Amount of deviation from path 

o Max deviation 

o Time in a tolerance band 

o Convergence/divergence 

b Similarity to reference path 

o Shape of deviation 

o Time significant deviation occurs 

o Frequency of significant deviations 

o Rate of error correction 

o Way error is corrected 

o Number of errors that occur simultaneously 

o Error in differential 

o Critical variable values exceeded 

o Time critical variables values are exceeded 

o Convergence/divergence to reference point on 

path trajectories 

o Shape of trajectory 

o Variable out of tolerance 

o Amount variable is out of tolerance 

o Time variable is out of tolerance 

o Number of errors in sequence 

o Number of critical errors in sequence 



Data Management 

Due to the great volume of data that must be handled by the processor, systematic data management 
is of great importance. This is basically a housekeeping operation which controls the coding of data and its 
efficient storage and retrieval. 

Data Smoothing 

Examination of recorded flight data shows occasional noise **glitches ' on the data samples. These 
glitches .occur at random times and must be removed prior to processing. Noise glitches are assumed to be 
pulses applied to the filtei^ that smooth data prior to sampling. Thus, the noise pulse appears as a pulse 
with an exponential^decay as shown in Figure 3. The resulting sampled values show a large sample to sample 
delta change between the samples before and after the noise pulse. 

Detection of the noise is accomplished by comparing the sample to sample (delta) change with a 
pre-established criterion value as follows: 

la.^^ -aJ < c, 1= 1,2,. . . n (1) 

where a^ is a sample value and c is a delta criterion value. If the inequality is not satisfied, a noise pulse is 
assumed to exist. 

Once a noise pulse is detected, the time duration of the disturbance must be determined. Experience 
has shown that the nominal disturbance duration can be expected to be .1 seconds (10 samples at a 
sampling rate of 100/sec.) for the recorded T-37 flight data. The duration of the disturbance is computed 
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by examining the value of future samples one-by-one to determine which sample value first falls within the 
acceptable region. The test is given by the inequality: 



la;^j - a, I <c*j, j=l,2,.... M 



(2) 



The smaUest value of j for which the above inec iality is true, is one greater than the number of samples 
included in the glitch. 

When the samples included in the noise glitch have been determined, those sample values are replaced 
by interpolated values. These values are determined by: 



(3) 



a..^j = a. + I j=l,2,... n-1 

where n is the value of j for which inequality 2 is true. 
Maneuver Sectoring 

a Introductioth Automatic performance assessment normally involves the computation of several 
different measures, each of which corresponds to different aspects of (or skills used in) the task of interest. 
In most flight maneuvers, the skUls required (and measured) vary from segment to segment of the 
maneuver. For example, different skills (and measures) are required during the downwind portion ot an 
approach and landing than are required during the turn to final or during the flare and touchdown. 
Therefore, it is necessary to segment the maneuver by idenrifying natural breakpoints which delineate 
portions requiring computation of different measures. Once these breakpoints have been identified 
algorithms and computer techniques are needed for automaticaUy detecting them on the basis of recorded 
pilot performance data. This secrion describes the development of such techniques for inclusion in the 
processor and their trial applicarion to several undergraduate pilot training flight maneuvers. It is 
appropriate to point out that automaric segmentarion of performance also has utility in a number of 
advanced simulator training capabOiries. For example, an increasing number of requirements for and 
applicarions of automaric malfunction insertion are emerging in recent and current flight simulator 
developments. To automarically insert a malfunction at the point in a mission that is realistic for the 
malfuncrion and at which the highest training value is expected, it is first necessary to automatically detect 
the desired point (thus, the utility of automatic segmentation). Other advanced training capabilities such as 
reiniriaUzation of the simulator and subsequent playback of a portion of the performance also can make use 
of segmentarion techniques (for automaricaUy detecring the point from which playback is desired). Finally, 
the distinct tr^nd toward the use of cathode ray tube (CRT) displays (rather than or in addirion to aircraft 
repeater instruments) at simulator instructor starions suggests another applicarion of automaric 
segmentarion. Present display techniques are to: (1) always display everything the instructor/operator may 
ever need to see during the enrire mission, or (2) allow various CRT "pages'' to be manually selected. The 
first technique is objectionable due to the number of displays required and subsequent load or the 
instructor/operator information sorting and processing requirements. The second is equally objecrionable 
due to the instructor/operator informarion retrieval load, Automaric segmentarion techniques could be 
used to assure that display contents always suit the instructor/operator needs based on what the student is 
practicing. The techniques described herein could be usefully employed for any of the above applicarions. 

b. Approach, The* approach was to develop techniques for generaring a mathemarical representarion 
of the state of pilot/aircraft performance which could be applied to any maneuver. Using this 
state-representarion, segmentation logic was developed for detecting specific states corresponding to the 
desired breakpoints within each maneuver. (The breakpoints themselves were idenrified largely on the basis 
of maneuver analyses performed as a part of other performance measurement studies (Connelly, Bourne, 
Lx)ental, Migliaccio, Burchick, & Knoop, 1974). This secrion describes the state representation techniques 
and the basic segmentation logic that was developed. 

(1) Maneuver State Representarion 

The technique for representing maneuver-performance states was to model significant aspects of the 
various performances using Boolean funcrions. The specific Boolean funcrions used differed from maneuver 
to maneuver as applied in various combinaUons to represent desired states.' However, since many of the 
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functions were found to be applicable to more than one maneuver, the approach used was to develop a 
universal function-set and associated Boolean notation which could be computer-implemented and, 
collectively, would satisfy state-representation requirements for all the maneuvers. 

The devised Boolean functions and the associated notations included indications that a specific 
condition (Boolean Function) is presently true or that the condition was previously true at least once/ 
Furthemiorc, functions and notations were developed to indicate when a condition is first true, last true 
(i,e„ when it becomes false), and the interval of time during which the condition is true. 

The notation used for representation of states is summarized in Table 2, As shown in the table, the 
Boolean notation "A = 1" is used to indicate that condition A is presently true, and "A = 0" is used to 
indicate that condition A is presently false, in addition, "A" indicates whether or not A has been true 
during the maneuver. Initially, A is set equal to zero; if condition A becomes true at least once during the 
maneuver, then A is set equal to 1 for the remainder of that maneuver. This provides Boolean notation with 
a "memory'^ and allows a logic function to be written in terms of present, as well as, previous events. 



Table 2, State Represeniation Notation 



Notation 



A= 1 


Condition A is presently true. 


A = 0 


Condition A is presently false. 


X = 0 


Condition A is not and has not been true 




during this maneuver. 


X=l 


Condition A is, or has been true during this 


t(A) 


maneuver. 


Time A became true. 


t(X) 


Time A first became true. 


t(S) 


Time A became false. 


\ A=X<z[ 


Defines logic variable as 




A=lifX<Z 




A-OifX>Z 



The time thatjevents take place is also important. Thus, t(A) represents the time that condition A 
became true, and t(A) is the time that condition A became false. This is aiustrated in Figure 4 where 
condition A is true for a period of time and then false. Note that the symbol t(A) iiidicates the time that 
condition A first became true and is always equal to some corresponding t(A). However t(A) itself may vary 
over the maneuver if the associated condition (A) changes from false to true more than once. 

The devised functions and notation can be used to detect the sequence in which events occur. 
Consider the time plots of two aircraft variables, pitch (0) and roll (0), shown in Figure 4. In plot 1 in the 
figure, pitch reaches zero first, whereas in plot 2 roll reaches zero first. In cases 1 and 3, accordingly, the 
Boolean conditions B and C indicate when the two variables of interest are zero. In case 2 and 4,' the 
Boolean conditions with "memory" (S and Q remain true for the subsequent time samples after they 'first 
become true. (In cases 5 through 8 and 9 through 12, AND/OR combinations of the Boolean variables are 
illustrated, respectively.) 

This notation provided a concise and easily appUed framework within which all desired states could 
be defined, logically manipulated, and tested. Although simple in appearance (Table 2), the notation is 
powerful enough for use in developing logic which detects the nature as well as the sequence of various 
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Figure 4. Examples of logic notation. 



19 



23 



_r 

_r 



JL 



Jl 



T 
F 
T 
F 

T 



T 
.F 

■T 

F 

"T 
F 

-I 

F 



pilot actions, as observed through their affect on aircraft performance. The following paragraphs describe 
the Boolean function set that was developed and the segmentation logic used for five fliglit maneuvers. 

(2) Segmentation Logic 

Table 3 lists the set of individual Boolean functions identified for use In developing the segmentation 
logic. The function numbers shown in Table 3 are used in Table 4 to sunmiarize the segmentation logic 
developed for each of five maneuvers. Note that only 17 of the Boolean functions were required for the five 
maneuvers that were investigated. Based on this 'Observation and the representativeness of the maneuvers 
tested, it appears that the Boolean function set is more than adequate for developing segmentation logic for 
conceivably any basic flight training maneuver. 

Tables 5 through 9 present the segmentation logic for each maneuver separately, including a 
descriptive title of each segment and the Boolean functions used for detecting the states determining the 
desired breakpoints between segments. 

c. Trial Applications. To perform Initial tests of the logic, sample pilot performance data recorded 
on a T-37B aircraft were used. (The basic data acquisition system is described In (Knoop & Welde, 1973). 
For this data collection effort, some revisions were incorporated including the addition of stick force 
sensors and the Increase of both range and reliability of aircraft attitude sensing. Revisions made to the 
original data acquisition system are documented in (Gregory & Cavanagh, 1973).) The performances were 
flown by instructor pilots who purposely demonstrated examples of how a novice might perform each 
maneuver as weD as examples of skilled maneuver performance. Four flights of each of the 5 maneuver 
types were used, two of which were rated excellent by the performing pilot and two rated poor. This 
provided examples of both performance extremes for testing the segmentation logic. 

(1) Cloverleaf 

The cloverleaf maneuver consists of a pattern of four consecutive loops, or leaves, all identical except 
for heading. For purposes of explanation, only the first leaf Is discussed. 

The leaf is begun after the start condition (level Hight) is satisfied. Figure 5 is the computer printout 
of one leaf of an excellent cloverleaf as processed by the logic (segmentation) program. Sector 2 begins 
when the pilot pitches up above T2. He then begins to roll (sector 3) until he reaches a maximum roll value 
(Cmi). Although excellent pilots generally roll to 180°, poor pilots often do not achieve 180°; hence, roll 
maximum is used to trigger the start of sector 4 because the logic must work on all types of flights. In 
sector 4, the pilot rolls back and pitches down until he reaches a minimum pitch (B^a)- Most pilots, 
regardless of their proficiency, begin to roll out before a pitch of -90° is attained; therefore, sector 5 
triggers on minimum pitch. The pilotMevels off his pitch (sector 6) prior to entering the next leaf, then 
begins the leaf by pitching up again (sector 2). 

(2) SphtS 

Tlie split s is an evasive type maneuver in which the pilot effects a 180° heading change by pitching . 
up, roiling over, and pulling out. The plot from the logic program is shown in Figure 6. Initially, sector 5 
triggered on pitch =- -90°. However, as with the cloverleaf, most pilots, excellent and poor alike, roll out as 
they pitch down and never reach >-90°. Therefore the condition was changed to minimum pitch (0m2)- 

(3) Lazy 8 

This maneuver consists of two halves, each of which are identical except for heading and direction of 
roll. The start condition (level flight) is a function of pitch and roll, while the subsequent sectors are 
identified solely on pitch angle. Figure 7 shows logic program output for the first half (sectors 1-5) of a 
lazy 8. 

(4) Normal Landing 

Tlie landing maneuver Is made up of five sectors. A sample logic program output is shown in Figure 8. 
On the sample flights examined in this study, the pilot did not lancl; instead he performed a touch and go 
maneuver. In either case, the maneuver is logically terminated when the touch down condition is detected. 
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Table 3. Boolean Functions 



No. 






Function 


No. 


Function 


0 


B. • ( 






22 


E2,= 'j Rs = Full left \ 


1 


B. ~\ 








C3 - < Ks - rull right ( 


2 


R = 




z4 


b4 - ]Ks = ReversRaJ 


Q 


' R — 








= ] E2E3 +E3EI } 


A 

'T 




d 


= +45° ± T4 } 


2j 


F, - jDt ~ Neutral ± S,f 




R. = ^ 




= -90° ± T5 1 


2o 


^2 -1 Dt - Full forward J- 


6 


R = 
ml 






Z / 


0, -| T - Idle J 


7 


R = 






ZOO 2 


ruture hxpansion 


8 


B 


_ 








9 


B . = 








ruture hxpansion 


10 


B c -- 








LZ T Lj 


11 


^ 1 






41 


Rl -PI 


12 


Co = 


't>>l2] 




dZ + UZ + y^j> 


13 


= 


4><h\ 




R 1 4. P 1 


1/1 
1^ 


p _ . 

C4- 




= +180°± I4} 


44 


CMl • Cl- 


15 






= +9o°±r5[ 


45 


CM2-CI 


16 






= -90° ± U 1 


46 


CiV13 • CI 


17 




i 




47 


C5 + C6 


18 


c„, = 

m z 






48 


E2 + E3 


19 


^m3 " 






49 


Future Expansion 


20 


D.=] 


As< Tasi } 


50 


Bl • CI Stop Condition 


21 


E, = • 


[Rs = Neutral ± R, \ 









Notes: lTAS,,Tn, andin, 


arc tolerance 


factors defined 


as follows for the five 


maneuvers: 




Cloverleaf 


Split S 


Lazy • 


Landing 


Barrel Roti 


TAS, 


0 


0 


0 


90 


0 


T, 


7 


7 


5 


4 


10 


T2 


5 


5 


8 


0 


2 


T3 


-5 


0 


-8 


0 


-5 




5 


5 


0 


0 


5 




5 


5 


0 


0 


5 


M 


5 


5 


5 


8 


5 




10 


9 


8 


10 


10 


*3 


-10 


-9 


-8 


-10 


-10 


*4 


5 


5 


5 


5 


5 


*S 


5 


5 


5 


5 


1 


^6 


5 


5 


5 


5 


1 



Computation of successive maximum or minimum values such asOfvjx requires that 

an intermediate null condition (Bi) be true. Thus the following sequences must occur in order for successive 
extreme values to be established: ^MX, B,,0mX+1) B,, 0mX+2) • • • 

Ri = general tolerance on rudder position 
$1 = general tolerance on stick position 

^0 - pitch 

0 - roll 

^Ml> 0MI " local maxima 

As -airspeed 

Rs - rudder position 

Dt - longituainal stick po.sitIon 

T - throttle position 

1 on 
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Table 4. Summary of Segmentation Logic 



Seg ments 







1 


2 


3 


4 


5 


6 


7 


8 


9 
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:i 12 13 14 


15 16 17 It 19 20 21 22 


Q overleaf 


1 


0 


2 


40 


17 


7 
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50 
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2 
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7 


1 50 
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2 40 17 


7 1 50 




4 
























2 40 17 7 1 50 


Splits 


1 


0 


2 


40 


3 


7 


50 














Lazy 8 


1 


0 


42 


6 


1 
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2 












43 


8 


1 


9 


50 






Normal 




























Landing 


1 


0 


40 


44 


40 


45 
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Barrel Roll 


1 


0 


3 


2 


47 


43 


50 















Table 5. Maneuver State Logic for a Clovei^eaf 



Sector 
Number 


Sector 
Name 


Condition 


Boolean 
Furjtion 


Function ^ 
Number 


1 


. Entry 


Pitch = 0°±T, and 


Bi-'C, ■ 


0 






RoU = 0°± I, * 






2 


Climb 


Pitch > 




2 


3 


Roll 


RoU<l3 (Left) or 


C2 +C3 


40 






RoU > (Right) 






4 


Pitch to 90° 


Roil 




17 


5 


PuU Thru 


Pitch = 0^2 




7 


6 


Final (entry to 


Pitch = 0°±T, ' 


B, ■ 


1 




next leaf) 







*From Table 3. 
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Table 6. Maneuver State Logic for a Split S 



Sector 


Sector 




Boolean 


Fu notion* 


Numb«r 


Name 


Condition 


Function 


Number 


I 


Start 


Pitch = 0° ±Tand 


B. • C. 


0 






Roll = 0°±I, ' 






2 


Entry 


Pitch >T2 




2 


3 


Inversion 


Roll > I2 or 


C2 +C3 


40 






RoU < I3 






4 


PiiU thru to 90° 


Pitch < T3 


B3 


3 


5 


PuU thru to 0° 






7 




Stop 


Pitch = 0° ±Ti" and 


B, - C, 


50 



Roll = 0° ± Tl 



*From Table 3. 



Table Z. Maneuver State Logic for a Lazy 8 



Sector 


Sector 




Boolean 


Function* 


Number 


Name 


Condition 


Function 


Number 


I 


Entry 


Pitch = 0°±T, and 


B, • C. ' 


0 






Roll = 0°± I, ' 






2 


1st Quarter 


Pitch >T2 or 


B2 + C2 + C3 


42 






RoU>l2 0r 










Roll < I3 






3 


2nd Quarter 


Pitch = ^^, 




6 


4 


3rd Quarter 


Pitch = 0°±T, ; 


B, 


1 


5 


4th Quarter 


Pitch = ^M2 


Bm2 


7 


6 


1st Quarter 


Pitch = 0° ±Ti or 


B, +C, 


43 






Roll=0°±li ' 






7 


2nd Quarter 


Pitch = ^M3 


Bm3 


8 


8 


3rd Quarter 


Pitch = 0°±T, 


B, 


1 


9 


4th Quarter 


Pitch=^M4 


Bm4 


9 


Stop 


End 


Pitch = 0° ±Ti and 


Bi ' 


50 



Roll = 0°±I, 

■*^From Tabic 3. 
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Table 8. Maneuver State Logic for a Normal Landing 



Sector 
Number 



Sector 
Name 



Condition 



Boolean 
Fu nction 



Function* 
Number 



1 

2 
3 
4 

5 



Stop 



Start 
Pitch Out 
Downwind 
Final Turn 
Final Approach 

(Touch Down or 
Touch and Go) 



Pitch = 0°±T, and 
RoU=0°±!, ' 
RoU>l2 or 
RoU < I3 
RoU-0°±I, 
after Roll =0mi 
Roll > I2 01 
Roll < I3 
RoU = 0°±I, 
after Roll = 0m2 
Airspeed < T^^^ 



C2 +C3 
C2 +C3 



0 
40 
44 
40 
45 

20 



*From Tabic 3. 



Table 9. Maneuver State Logic for a Barrel Roll 



Sector Sector Boolean Function"^ 

Number Name Condition Function Number 



1 


Start 


Pitch =0°±Ti and 
Roll-0''±li 


B, • C, * 


0 


2 


Entry 


Pitch <T3 


B3 


3 


3 


1st Quarter 


Pitch > T2 


B2 


2 


4 


2nd Quarter 


Roll = 90° ± Is or 
Roll = ^90° ± k 




47 


5 


3rd Quarter 


Roll= 180° ±14 


C4 


14 


6 


4th Quarter 


Roll = -90° ± \e or 
Roll = 90° ± Is 


Cs + Ce 


47 


7 


End 


Pitch =0°±T| or 
RoU=0°±l, ' 


B, +Ci 


43 


Stop 




Pitch = 0°±T, and 
Roll=0°±lt ' 


B, ^ C, ' 


50 



*Scc Tabic 3. 
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(5) Barrel RoU 

This is an acrobatic maneuver in which the pUot rolls througli 360°. A sample logic program plot 
appears in Figure 9. The sample data indicates that inexperienced pilots have difficulty attaining level fliglU 
(Bi • Ci)at the end of the 4th quarter (sector 6); consequently, sector 6 contained a long tail of data not 
representative of the specific pilot task we were trying to isolate. A seventh sector was added to allow 
termination of sector 6 if either pitch or roll is level (Bi + Ci), The maneuver terminates when the pilot 
attains level flight. 

Regression Analysis 

a. IntroductiOfL A major purpose of the regression analysis is to generate reference functions which 
are representative of excellent performances. These reference functions are automatically generated by the 
processor for use in deriving performance measures. 

A number of reference functions are constructed for each sector of each maneuver type. Each 
function is a mathematical representation of certain parameter relationships characteristic of that sector. 
Deviations of an actual flight from this function are computed. A standard set of operations on these 
deviations are performed, and results are tested for performance discrimination content. The techniques of 
measuring and interpreting these deviations are discussed in detail in later sections of this report. 

A useful reference function must be consistent, in that it produces small deviations with data from 
excellent performances; at the same time, it must be able to provide discrimination in tests among various 
performance levels, in the processor, reference functions are generated, then tested for consistency and 
discrimination capability. This procedure is discussed in the following sections. 

b. Theory. Several reference functions are generated for each sector of each maneuver by performing 
a least squares regression analysis on selected skilled performance data for the specified sector. In our initial 
analysis, four candidate reference functions are generated for each maneuver sector by using data from two 
available excellent-rated flights of each maneuver type. The technique is illustrated in Figure 10. 

(1) Regression Computarion Method 

In applications, many sample flights of several performance categories will be used to form the 
reference functions and update Uiem. The data are initially arranged on tape by maneuver type and it is not 
leasible to store all data or to reread the tape for each maneuver sector. Therefore, it is desirable to use a 
technique which allows updating of the regression coefficients Without having to store all previous raw data. 
Tlie method, discussed in detail in Connelly et al., (1969, pp. 179481), represents the data in a compact 
summary form. Briefly, the problem is stated as: 

N 

Y = A + 2 B X (3) 

j=l 

where Y is a factor of interest (dependent variable) and Xj is a combination of the system variables 
(independent variables). Given T samples or experiments, the method of least squares minimizes P: 



ERLC 



T N 
P = 2 (-Yi + A+ 2 BX )^ 
. i=l j=l 

and gives solution values for the coefficients: 

T T N T T T N 

-2 Y,x^,+ 5) 2 BX.x^i+ 2 ^kiO/T2 y,-i/t2 2 bx.) = o 

i=l i=l j=l i=l i=l i=lj=l 
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(4) 
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where 



K = 1,N 

T T N 

i=l i=l j=l 
Defining the following sums: 



(5) 
(6) 



SUMYX(K) = 2 Y. X^. 
i=l 



(7) 



SUMXX(J,K)= 2 X X^. 
i=l 



(8) 



SUMX(K)= 2 \i 
i=l 



(9) 



SUMY = 2 Y. 
i=l 

allows representation of the problem as; 

N 



(10) 



N 



0 = -SUMYX(K)+ 2 BjSUMXX(J,K) + SUMY*SUMX(K)/T-SUMX(K)/T* J) B. SUMX(J) 
H . j=l ' 



where 

K= 1,N 

Solving for the B's is done via a matrix approach: 
3(1) 

Let j5 = B(2) 



(11) 



(12) 



R = 



L B(N). 



SUMYX(1) - SUMY * SUMX(1)/T 
SUMYX(2) - SUMY * SUMX(2)/T 



. SUMYX(N)- SUMY * SUMX(N)/T 



(13) 
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Q(J,K) = SUMXX(JJC) - SUMX(K) * SUMX(J)/T 
(known as correlation coefficients) 



(14) 



SUM = 



0(1,1) 
0(1,2) 



0(2, 1) 
0(2,2) 



0^, 1) 
Q(N, 2) 



,0(1, N) 0(2, N) 0(N,N). 
(an NxN matrix) 

The problem is now written as: 

SUjM-B'=R 

and the solution^j^j 
B = SUM-^'R 

Now A can be computed from equation 6: 

N 

A = 1/T ^SUMY - 2 B(J) SUMX(J) ^ 

For ease in programming, the matrices SUM and R can be further broken down: 
SUMXX(1, 1) SUMXX(2, 1) • • • SUMXX(N, 1)" 

SUMXX = 

SUMXX(1 ,N) SUMXX(2,N) • • • SUMXX(N, N) 
SUMXJ= [SUMX(I) SUMX(2) . . . SUMX(N)] 



SUMXK = 



SUMX(l) 
SUMX(2) 



SUMX(N) 

Then SUM = SUMXX - 1/T * SUMXK * SUMXJ 
SUMYX(1)" 



Let 



SUMYX = 



_SUMYX(N). 

VaenR= SUMYX - SUMY/T * SUMXK 



(15) 



(16) 



(17) 



(18) 



(19) 



(20) 



(21) 



(22) 



(23) 



(24) 
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Therefore 

B= [SUMXX - IIT SUMXK * SUMXJ • [SUMYX - SUMY IT ■ SUMXK] 



(25) 



This technique provides, for efficient computer usage by compact storage of all previous data that is ri'eeded 
for updating of the regression coefficients. 

(2) Candidate Reference Functions 

In the processor, rp.ference functions are based on data from several excellent maneuvers. Initially, 
one excellent maneuver sector is read, and a least squares regression is performed on it. The data are read 
sector by sector and maneuver by maneuver from tape; when another excellent maneuver sector is 
encountered it is used to update the regression. 

Four candidate reference functions are generated for each maneuver sector: 

^Max ' ^Max) - Bo + B, X + B,X^ + Ba^M + B40m 

The above variables are: = maximum pitch in the sector, 0m maxiumum roll, h = altitude, 4/ = heading, 
AS = airspeed, d = pitch, and the "A" means "estimate", X, the independent variable, can be roll, pitch or 
normalized time. Roll (cp) is always selected first if roll is monotonic in that sector. If roll data is not 
monotonic over the sector, pitch (d) is selected; and the fourth reference equation then becomes: 

0 = Bo + B,0 + B,0^ +B3 0M + B4 

If neither pitch nor roll is monotonic, normalized time (t) is selected as the independent variable, and in 
that case the fourth reference function can have either 6? or 0 as the dependent variable. 

(3) Performance Discrimination 

The purnose of a reference function is to specify a standard flight path for use in developing 
performance ^me^ures. It is necessary, therefore, ftJTITrefence function to give consistent results for all 
excellent performances, and at the same time provide a basis for discriminating performance that are other 
than excellent. The measure used for a preliminary test is tlie mean absolute residual error: 

e = 1/T 2 lY.-Y.; 
i=l 

wher^T is the number of samples in the maneuver sector; Y{ is the actual value of the dependent variable; 
and Y{ is the prediction of Yj by the reference function. The value of e gives an indication of how much a 
sample flight deviates from the reference function. If the reference function possesses good performance 
discrimination capabilities, then e would be expected to be small for excellent flights, and to increase as the 
performance level worsens. A graphical interpretation of e appears in Figure 11. 

In the processor, candidate refe rence functions are generated from several excellent maneuvers. These 
maneuvers can be considered to be a "training set" for the processor and this set possesses some mean 
residual error, ej, with respect to each reference function. As a test of the consistency and discrimination 
ability of the reference function, "test sets" are formed consisting of one set of excellent maneuvers not 
included in the training set, and one set of "poor" maneuvers. The test sets also have a mean residual error, 
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Table 10, Independent Variable Selected for Each Maneuver Sector 



Sector 



Maneuver Type 1 2 3 4 



Cloverleaf 


Time 


Pitch 


RoU 


Pitch 


Pitch 


Splits 


Time 


Time 


RoU . 


Roll 


Pitch 


Lazy 8 


Time 


Roll 


Roll 


Pitch 


Roll 


Normal Landing 


Time 


Time 


Time 


Time 


Time 


Barrel Roll 


Time 


Time 


Roll 


RoU 


Roll 



Table 11. Regresion Analysis Results for Sector 1 of the Cloverleaf 



Errors 



Dependent 




Exceiienf 






Poor 


Variable 




125(L) 


1 24 (R) 


127(L) 


12t(R) 


h 


25.44 


25.05 


25,73 


3794.50 


3478.60 




0.49 


.09 


0.79 


.68 


2.79 


AS 


1.11 


1.14 


1,09 


3122,40 


2730.00 


6 


0.87 


.79 


.93 


170,35 


146.99 


<t> 


0.34 


.36 


,32 


17,32 


15.76 


#/samples 




17 


23 










-7.0 


-7,7 


5.4 


6.3 






6.0 . 


-1.6 


-A 


3.1 




FUNCTIONAL FORM: DV = Bq 


+ Bit V B^t^ + B3 


Om + B4 0M 








Reference Functions 






Variable 


Bo 


B, 


B2 


B3 


B4 


h 


2240.00 


-631.20 


220.50 


318.10 


-141,80 




-.40 


.44 


0.08 


-0.02 


-.29 


AS 


1890.00 


28.70 


-11.78 


254.90 


-175,70 


Q 


' '90.58 


—23.55 


30.65 


14.05 


-8,48 


0 


10.22 


-2.81 


2.56 


1.37 


-0,09 
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€s, with respect to the reference function. If the reference function rx a suitable basis for performance 
discrimination, then 6$ should be approximately equal to ex for a test set comprising excellent maneuvers, 
and significantly greater than for a test set made up of poor maneuvers, A simple test is to compare the 
distribution of residuals, e^i, from the training set to the distribution of residuals, esi, from' the test set via 
the rank sum statistic (see Rank Sum Statistic section). This statistic tests the hypothesis that the two sets 
(training and lest) are equal; i.e., that the reference function in question produces similar results for two 
different sets of maneuvers. The rank sum test is used because it requires no knowledge about the 
distribution of residuals, and it can compare two data sets of different lengths. 

c. Results. A total of 19 maneuvers were run. The independent variables chosen for each maneuver 
sector, as discussed in Regression Analysis section, are shown in Table 10. Table 1 1 shows the actual 
reference functions for sector 1 of a cloverleaf. Tlie functions are written in the bottom table below the 
functional form. The residual errors (£) appear in the top table. Column 1 is the combined error of the two 
excellent maneuvers and is computed as follows: 

^c"" Ni +N2 

where Nj and N2 are the number of samples in the two excellent maneuvers and and ^2 ^re their 
associated errors. Columns 4 and 5 show the errors for the two poor maneuvers. 

Regression functions for other cloverleaf sectors and other maneuvers appear in Appendix A. 
(The three digit numbers for the maneuvers were assigned for maneuver identification, A summary of 
maneuvers used is shown in Table 12.) Some sectors contained too few data points to obtain a significant 
regression; consequently, no data appears for them. 



Table 12. Maneuver Identification Code for 
Computer Processor Printouts 



Maneuver 


Mode 


Proficiency Rating 


ID Code 


Cloverleaf 


Riglit 


Excellent ^ 


124 




Left ' 


Excellent 


125 




Left 


Poor 


127 




Riglit 


Poor 


- 128 


Split S 


Left 


Excellent 


126 




Left 


Poor 


129 




Left 


Poor 


130 


Lazy 8 


Right 


Good Pius 


102 




Right 


Poor 


106 




Left 


Excellent 


108 




Left 


Poor 


113 


Normal Landing 




Excellent 


121 






Excellent 


122 






Poor 


123 






Poor 


124 


Barrel Roll 


Left 


Excellent 


101 




Left 


Excellent 


107 




Left 


Poor 


110 




Left 


Poor 


114' 



( , . 



37 

41 



On certain maneuvers, very small differences in ^Max and 0Max seem to cause large differences in the 
residual errors. For example, in sector 6 of the normal landings (Appendix A), the coefficient of 0Max for 
the heading regression is extremely large (B4 = -7990). Although 0m of maneuver 124 differs from 0m of 
the excellents by only about two degrees, the difference in 6's is huge. Based on tlie processing of four 
maneuvers, it would appear that an excellent maneuver with a slightly different 0m could significantly 
increase the E*s for the excellent maneuvers. 

Further processing is required to determine if th*s is true in general. If so, one alternative would be to 
change the form of the regression; e.g., Y = Bq + B, X f 82 . However, lacking the necessary data to test 
many more maneuvers, as required, the regression functions in the processor were implemented as 
documented in this report. 

Adaptive Math Models 

This section describes the experimental techniques for generating candidate measures for subsequent 
validation-testing. The models for so doing are called adaptive mathematical models because the candidate 
measures which they generate are derived recursively and adaptively in accordance with the success 
encountered with various measure-types. Since much of the underlying mathematics has been documented 
in earlier referenced studies, emphasis here will be on a brief description of each model and those areas 
where refinements were incorporated as a part of this study. 

The purpose of the adaptive math models (AMM) is to systematically search Boolean time sequences 
(BTS) for various characteristics and determine if the characteristics are related to performance 
measurement. A block diagram of this process is shown in Figure 12. Smoothed flight data is directed to 
Boolean logic which processes the data and develops Boolean functions designed to succinctly represent 
critical performance-related information contained in the data. The output from the Boolean logic is a set 
of Boolean time sequences which are directed to three processes: relative, absolute, and state transfer. Each 
of these processes searches for different types of measures, as discussed in the Data Smoothing section. The 
processes are two-step operations in which first, the Boolean time sequences are systematically searched for 
characteristics and relationsliips potentially related to performance measurement. When useful 
characteristics are detected, a test is conducted to deterfnine'their significance to measurement. Finally, the 
outputs of each of the three processes, which are intermediate performance measures, are combined in a 
weighted sum to provide an overall performance measure for evaluation. ~ — 

Tu establish the notation used in following sections, consider a Boolean time sequence where a single 
bit of the sequence is represented by BTSjjK The first subscript (i) represents the Boolean function which 
generates the BTS, and the second subscript (j) idenrifies the jth element of that sequence. Thus, Boolean 
time sequence i is given by: 

BTS.^,j = l,M. 

where is the number of elements in the sequence. The superscript (k) is used to indicate the flight 
event or flight maneuver number associated with the Boolean sequence. It is seen that is a function of k 
only and not i, because every Boolean sequence generated with data from flight event k has the same 
length. When reference is made to a total BTS for a specified flight event, the notation BTS^. is used. 

a. Boolean Function Data Representation. A special transformation of the data is performed to 
simplify its analysis and to permit the user to interact with the processor by adding to it his knowledge of 
the problem. The transformation results in representation of the data in the form of BTS produced by 
applying a sequence of performance demonstration data samples to Boolean functions (BF). Two types of 
BF are constructed and will be discussed in turn: 
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• Functions representing the raw data itself, 'as expressed in terms of discretized deviations from 
criterion (or reference) functions; and 

• Functions which are demanded by the user and reflect liis knowledge about features of the 
performance that are considered by him to be relevant to measurement. 

(1) Functions Representing the Raw Data 

Without relying on guidance from the user, BF are constructed by modeling the deviation of the 
various parameters from standard profiles or time-histories. This is accomplished by, first, spanning the 
envelope of performance with amplitude test bands, as illustrated in Figure 13. The test bands (labeled 1 
through 5 in Figure 13) are determined empirically and consist of multiples of standard deviations around 
the reference function, computed using performances of skilled subjects. BF are constructed to represent 
activity in each test band. The BF are set true only when the actual performance data sample is within the 
limits of the respective test bands. 

(2) Functions Demanded by the User 

The user may construct special BF by asking pertinent yes-or-no questions about the performance. 
The answers (1 or 0) then form the values of the associated BTS. For instance^ the user may have reason to 
believe that whether or not a pilot's turns are consistently coordinated is particularly relevant to measuring 
his performance on a given maneuver. Therefore, he may ask whether 

Roll = f(pitch, rate-of'tum, . . .) ± 5 

where the function f is designed to model a coordinated turn. If indeed this information is relevant to 
measurement, then the level of activity (percentage of time true) of the associated BTS would probably be 
a good performance measure. 

As a second example, there may be reason to believe that performance at critical points in the 
maneuver is particularly relevant. To augment the processor with this information, the user might pose the 
question, "is the performance currently at a critical point (like Pitch = Max Pitch ± 10°)?'' The resulting BF 
then identifies that point in the maneuver. In this particular case, the associated BTS itself may not be 
relevant; rather, its logical relationship with other BTS (i.e.; what is happening at the critical point) would 
probably be of most value. 

b. Absolute Measures. The absolute computation mechanism consists of a correlation of each BTS- 
against a fixed set of functions or sequences (MacDonald Codes). This results in the transformation of a 
long sequence (BTS) into a new set of non-Boolean variables which in turn can be examined to .determine if 
they are relevant to performance evaluation. 

Correlation against an absolute reference allows a search for measurement-significance of particular 
sequences or patterns as they are generated by the Boolean functions. If it is found that some BTS pattern 
is likely to be predictive of skilled operator performance, this information can serve as a basis for 
specification of automatic scoring systems as well as provide clues about the operator techniques used in 
acliieving superior performance. The absolute measure also allows analysis employing multiple BTS as well 
as a single BTS via a regression computation. This provides the tools required for a systematic study of 
which Boolean function and combinations thereof are relevant to measurement. 

The absolute computation is defined as: 

= ^ 1 BTS.'[xZ.,N2 -Ni<M. 

N2 -N, j^j^^ . 0 0' ^ ^ k 

where Zq is an element in a reference sequence. The subscript (f) indicates which reference sequence is 
being used. Note that the summation is not conducted over the total length of the BTS; rather, it is 
computed over a short interval of the BTS. There are two factors that lead to this approach. First, every 
performance does not require the same length of time and as a resuh is not a constant. Thus, the 
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summation must not be over an interval greater than the smallest value of M|^. Second, and far more 
important, is that the set of reverences fj should be a set of orthogonal binary sequences in order io have an 
efficient reference set. (It can be shown that there are N such sequences N elements long; i.e., fj is a square 
matrix.) The absolute measure can also operate on threshold variables (see Adaptive Mathametrical Models 
section) instead of the BTS. In that case, the computation is defined as: 

N, 

The process implemented in the computer program uses a Hadamard transform producing coefficients 
from all reference sequences in one efficient operation rather than N operations. However, the process is 
described by the equivalent correlation operation with a reference sequence. 

The length of the reference sequence (and therefore the number of reference sequences) is taken as a 
varaible 2^, M = 2, 3, 4, 5, 6, in order to facilitate generation of the Hadamard transform. As stated 
previously, the reference sequence length must be adjusted to accommodate the length of the BTS; 
however, the length of the BTS is not the only factor of interest. The optimum length of the reference 
sequence required to produce sensitive performance measures is not known. It is known that the BTS 
pattern can be searched in several ways. Two possible ways are shown in Figure 14. Method A shown in the 
figure requires a correlation of the 4 bit reference sequence to 4 bits (in general 2M)of the BTS followed 
by correlation of the reference sequence with BTS bits 5-8, etc., until all BTS bits have been processed.. 
Since there are 4 reference sequences (of 4 bits each) the process is a multi-pass operation. The equivalent 
transform operation requires one pass. It can be shown that the values of the 4 coefficients for each shift 
uniquely specify the BTS and no information is lost by the correlation (transform) operation. (Preservation 
of information may or may not be necessary or a sufficient requirement in performance measurement. In 
fact, it is easily seen that performance measurement is an operation in which information is discarded 
systematically, thus reducing a great volume of data to a few variable values representing performance.) 

Method B employs a correlation (transform) operation followed by a shift of one bit, followed by a 
second correlation, etc. This method allows examination of each sequence of 2^ bits and for that reason is 
preferred. Various- length reference sequences can be processed without risk of an incomplete correlation at 
the end, due to a BTS length not equal to a multiple of 2"^ . Thus, the processor is designed to employ refer- 
ence lengths 2^ , M = 2, 3, 4, 5, 6, and use the shift pattern shown as Method B in Figure 14. 

Each correlation operation produces one value of the correlation coefficient and there arc N + 1 
correlations, where N is the number of shifts. If the BTS has L bits and the length of the reference sequence 
is 2^^, there are L~2^ + 1 correlation coefficients (C) for each BTS and reference sequence combination. 

Detection of "patterns'' in each BTS is accompHshed by analysis of the distribution of the correlation, 
coefficient (C) values obtained from each channel (BTS and reference sequence combination). A 
fundamental question is determining the C distribution that might result from a random BTS (i.e., without 
consistent patterns). Consider a random BTS where ^ach bit of the sequence has a probability of 0.5 of 
being a 1 or —1. This population has a mean and variance of: 



a^= (-1)^ X.5 + P X.5 = I 

The correlation operation can be considered as a summation of N elements of that population and the 
distribution of summation has a mean and variance 

a^' = Na^ = N 
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The distribution of a^^ is of specific interest since a computed sample variance from flight data is to be 
used to detect the existence of data patterns. Thus, the .90 and .95 probability points of the cumulative 
distribution for are desired for an automatic decision threshold so that the number of false pattern 
detections (Type I error) can be controlled. Analysis of this problem resulted in the identification of 
appropriate decision thresholds for incorporation in the processor, 

c. Method of Constructing System Stares for State Transfer Measures. EVevious efforts in 
constructing Boolean functions for the purpose of proficiency measurement, employed one or more 
threshold states- for each flight variable (Connelly et al,, 1969 & 1971). It was observed in those studies that 
improved results could be obtained using a new variable derived from deviations from reference functions. 
For example, in the lazy 8 maneuver, a function relating pitch and roll angles was used to provide a 
reference. Threshold states were constructed as Ka displacements (i.e., mulriples of standard deviations) 
from this reference function. These threshold states, sequences of threshold . states, and patterns of 
threshold states are used to obtain estimates of the system performance. 

While this method works well, it is desirable to extend the method such that threshold states for more 
than one reference function can be considered collectively. For example, a state transirion measure should 
be more effective where the states reflect collective deviations from m.ore than one reference. However, the 
number of combinations of threshold states can be large which leads to problems in computation, data 
storage, and data collection. 

It is possible to combine several threshold state combinations into performance states with what is 
believed to be a reasonable way. Benefits from such an approach include simplified computarion while 
maintaining a "physicaP' interpretation of the performance states. It is necessary to compare each sector of 
actual data to this reference path to obtain an indication of how much deviarion exists between the two 
(Figure 15). Instead of listing the sequence of the residuals (e^ = di —^\) for each sample, a new sequence 
can be written as follows. If le^l is less than la, define = 0. If Cj is between la and 2a, let = 1, etc. 
Now, the sequence of residuals is reduced to a sequence which contains values 0-4 which are states. The 
sequence can be considered as a function state since it shows the progression of states followed by the 
flight. This process can be generalized as follows: 

Consider a set of reference functions as follows: 

6^= (0,0Max,0Max) 

AS'= f^g (0,0 Max, 0Max) 

fj^ (0, 0Max, 0Max) 

^ = f (0, 0Max, 0Max) 

where 

0 is roll angle 
d is pitch angle 
AS is airspeed 
h is altitude 
i// is heading 

'x is the estimate of variable X. 
The error function for 0 is given by: 
= e - (q=Eq (d, 0, d Max, 0Max) 

M 
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5 10 t 

Sequence of residuals: | , e^, e^, , ... | 

Sequen-ce R : | 1, i, i, 2, 2, 3, 3, 4, 4, 3, 1, ... | 
Booiean Time Sequence: 

(0 R < 2 
BTS = < 

" 1 R > 2 

= I 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 
F/;gwre 15. DeviationJ)etween actual data and reference path. 
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where the value of EQq is one sigma deviation function for the distribution of 0 given (0, 0Max, 0Max), 
In a similar way, additional error and one sigma deviation functions can be formed 
E^5 (AS,0,0Max,0Max) 
^Asa ^Max, 0Max) 
\ (h, 0, 0Max, 0Max) 
E^^(0,0Max,0Max) 
(i//, 0,0 Max, 0Max) 

Define threshold variables as: 
=0 when [ E^Kk E^^ 

=1 whenFCE^^ <|E^|<2k E^^ 
R^ =2 when2kE^^<|E^r<3k E^^ 
R^ = 3 when 3kE^^ < |E^|< 4k E^^ 
R^ =4 when4k£^^<|E^| 
where X = 0, AS, h, i//. 

A system state can be formulated as: 

Thus, S values range from 0 lo 16. The following diagram illustrates the translation between threshold 
regions activity and the system state representation. 
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This method provides a summary perfomiance state wliich renders several individual threshold states 
equivalent. The method allows a compact 17 state representation of a system that contains many more 
states, 

Fn summary, there are several types of system states available. These are: 

• Binary Threshold states 

A binary threshold state is defined by a binary valued function indicating if the present BTS 
sample (residual) exceeds a specified level, i.e., 
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BTSy = lifR;j>k. 
= OifR..<k. 



where R.j is the threshold variable and k is Uie associated threshold factor. 

• Threshold states 

A threshold state is the value of the threshold variable R^^ = 0, 1, 2, . . , 

• System states 

A performance state is the value of the system variable(s) given by the sum of the threshold 
variables S = + R^g + Rj, 

The BTS is a sequence with 1/0 valued elements. In a strict sense, only binary threshold states can be 
represented in a BTS; however, for computation of transition measures threshold and system variables are 
useful. In the state transfer computational mechanism, the multi-valued variables R and S are used to 
replace the binary valued BTS. 

d. State Transfer Measures. The state transfer computation mechanism is a means for determining if 
performance (or score) information is related to the sequence of operator actions. It may be that operator's 
performance is partially or totally a function of how he corrects for errors, where he may or may not have 
caused the errors initially. 

In order to implement this computation and also provide a convenient means for compactly 
representing long Boolean time sequences, a transition matrix is formed which identifies how the sequence 
moves from state to state. Also, a composite transition matrix can be formed to represent transition 
patterns from all demonstration data sets (DDS) of a given performance level. 

A Boolean state is defined by the set of binary values associated with the set of selected BTS as 
described in the previous section. Each DDS can be viewed as a sequence of Boolean states or, alternatively, 
as a sequence of state transitions termed "transtates." The state transition measure seeks to relate the 
frequency of use of each transtate to performance measurement. This is accompHshed by associating with 
each transtate a score value stored in an incremental score matrix (ISM). A performance measure value is 
produced by summing the score values from the element of the ISM corresponding to each transtate used in 
the DDS. The final performance measure value is obtained by dividing the sum by the number of 
transitions. The transition matrix is formed using the system state values 

S = 0, 1,2, . . . , 16 

A 17 X 17 transition matrix is required to store all transition probabilities. 

If we assume that the DDS state sequence can be described as a Markov Process, the sequence can be 
represented by its transition matrix. Under this assumption, the performance measure can be computed in 
another way. A fundamental theorem for Markov Chain processes states that if tTq is the initial probability 
vector (probability distribution density), then tt^ (the probabiHty vector after n trials) is given by 

7r =7r T" (1) 

o o 

where T is the process transition matrix. Proof of the results given here can be found in Connelly et al., 
1969. 

Now, we assume that the process is a regular Markov process. Such a process is identified by a 
transition matrix (T) where for some value of n, T" has no zero elements. This implies that the..system 
could be in any state after N trials, independent of the initial state. . 



O 47 

ERJC 51 



The assumption that the system is a regular Markov process lets us state that there exists a unique 
probability vector (a) such that 



as n -> 0°, limit tt^^ = a (^) 

Elements of this probability vector correspond to the probability of the system being in the associated 
state. ' 

Furthermore, it is seen that 

aT = a (3) 

must hold due to Equations (1) and (2). This limiting probability vector gives the first state distribution 
desired. Note that a is the probability distribution of finding the process in each state, given that we have 
not observed the process previously. 

The distribution of transtates can be determined by imagining an ensemble of many adjustment 
systems with first states distributed according to a.. The second states for each system are determined 
according to the transition matrix T. This yields new states, also with a distribution a (according to 
Equation 3). The probability that transition i-j is used in the operation is the probability of the transition 
(Tjj) given the first state times the probability of being in state i (aj). 

Let a! be an N by N matrix with zero value elements off the main diagonal. Also, the elements {p!\\) 
of a' are given by 

a'..=a. 

11 1 

A new N X N matrix (D) is defined as 
D = a'T 

Elements of D (aj Ty) are the probabilities of the system being in each transtate assuming the first (or any 
other) state is not known. 

The probability matrix (D) can be used to establish the equivalent population statistics. Elements of 
D (i,j) may be considered normalized weighting factors, and elements of ISM (i,j) provide the population 
values. The population mean (P) is 

N N . 
P= S 2. D(ij)lSM(iJ) 

i=i ri 

P is equivalent to the performance measure computed from state transition as described previously, in 
addition, we now have the tools for computing values for the ISM. The method is to form a representative 
transition matrix for. two or more performance levels of the DDS. In this way, one transition matrix 
represents excellent performance and another represents another performance level, etc. Once these 
composite transition matrices are available, the elements of ISM can be adjusted (trained) to improve the 
performance measure discrimination capability between (or among) the demonstrated performance 
categories. The method is to sequentially adjust the ISM using one transition matrix at a time and to 
continue the process until a measure with stable discrimination capability is obtained. There are alternative 
methods of adjusting the ISM, but this iterative method converges rapidly and allows introduction of new 
data as it is obtained. The iterative method requires computation of the amount each element in ISM 
should be changed in order to modify the score (measure value) by a specified amount. 

The probability matrix D (i j) can be used to compute the expected value of score change by means 
of the adjustment process. For each transition, the probability that transtate (ij) is used (assuming we do 
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not know the present state) is D (ij). Thus, the expected change in the associated incremental score, ISM 
(i j), at each transition is the product 



A-D(ij) 

where A is the amount added to the ISM increment. Since no transitions are used in the sequence, the total 
expected change is given by: 

n2 • A • D(i,j) 

If we use ISM (ij)' to represent the expected updated incremental score matrix, we see that: 

ISM(iJ)' = lSM(ij) + n2 - A- D(iJ) 
The expected mean score is 

^= 1 2 D(i j) • lSM(i j) + n, • A • (D(i j))^ 
i j 

and the expected change in mean score (CM) is 

CM = n, • A • Z Z (D(i j))^ 
i j 

The summation term can be considered as the system gain G, such that 

G=XSP(ij))' 
i j 

(Note that G will be less than one.) 

Thus, the amount that must be added to each element of ISM is^n2 - A - D(i j) in order to change the 
measure value by amount CM. ^ 

e. Relative Measures, The relative computation technique operates on up to four Boolean functions 
simultaneously to determine if logical relationships exist and, if so, how the relationships are associated 
with performance. Thus, as opposed to the absolute computation where operations are performed on a 
single BTS, the relative computational technique uses a "trainable logic" concept to detect possible 
relationships among BTS. The approach is to select a set of base BF channels and form all combinations of 
these sequences for each data sample. For example, if three base BF channels are selected, eight 
combinations can be produced. Only one combination is true for each sample. Correlation of one 
combination (say combination j) with an additional BF channel (say BFi) yields the conditional probability 
that BFj is true given that combination] is true. 

Each combination can be expressed as : 

c G)= (^BTS^. bts^Kbts^^ 

where the first subscript indicates the BTS, and the superscript indicates the DDS. The second subscript (j) 
indicates the element in the sequence. The AND operation is conducted on a bit-by-bit basis. Therefore, the 
combination C^Q) has a binary value corresponding to each element in the BTS; i.e., Cj„(j) is a Boolean 
sequence itself. 

As an example, we determine combination 5. It is convenient to represent m (or 5 in this case) by its 
binary form (101). Thus, using the binary form of m to code the combination, we find 
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C5(j) = BTSH BTS.^. BTS . 
^^'^ cj bj aj ' 

where BTS is the NOT function. Note that the second subscript need not be the same for each term; i.e., we 
can compute using relative time positioning by selecting (for example) second subscripts j+3,j-l). 
While this flexibility is available it has not been extensively explored to date. 

The analysis consists of the following operations. First, a number, SUMn, is computed which is a 
count of the number of times each combination occurs during the flight event under study. Thus, 

^ j=l 

A second count is formed which represents the number of times a BTS variable is true, given that 
combination is true. Thus, 

SUM^= E C^G)BTS^. 

Normally, the first subscript e would have a different value than those of the base variables. Next, the 
conditional probability that BTS^j is true given that combination n has occurred is determined as 

SUM^ h 

P,(n)= 

SUM^ 

This conditional probability identifies the relationship between each combination of the base variables and 
the predicted variable. These conditional probabilities are candidate performance measurement variables 
and are tested for validity, as described in Validation Tests section, 

f. Summary Description of Measures, Figure 16 illustrates a 2-dimensional state-space defmed by roll 
and pitch angles, and the approximate trajectory outline of one quarter of a lazy 8 maneuver. The small 
arrows in the figure depict alternative directions in which a roll/pitch trajectory might move in a given 
performance. The State Transfer type of measure is based on probabilistic assessments of this direction of 
movement. To compute the probability values, the state-space is gridded into discrete states. (For instance, 
each cell or rectangle in Figure 16 may be considered an individual state.) By computing the frequency with 
which each state is acquired and the state sequence, the probability of transfer from state to state is 
calculated. 

The State Transfer computational mechanism can operate on up to four flight variables at a time. To 
minimize the number of states to be handled simultaneously and the associated computational complexity 
of the problem, threshold states are used which represent the sum of the deviation units from each criterion 
function. For example, in Figure 16 the shaded cells might be recorded as threshold state number 1 
(depending on computed performance variance and resulting cell sizes used to model each performance), 
because they are located one deviation unit from the reference trajectory. This state representation not 
only reduces problem complexity, but permits ready interpretation and assessment of divergence from or 
convergence on criterion terminal performance. 

The Relative measure is based on conditional probabilities of various states being acquired 
simultaneously with the acquisition of other states. For instance, consider the user-defined BTF's of (1) 
Pitch = maximum pitch ±Ai , and (2) airspeed = Ajc ± An , where Ajc is a criterion airspeed value. Analyzing 
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Roll 

f'li(urc 16. Representative roll/pitch state space. 



these BTF's along with the information in Figure 16, the Relative computational mechanism would 
examine as a potential measure the probability that pitch i^; maximum whenever airspeed = Aj<; and state X 
in Figure 16 is active. (Tliis is precisely a significant factor of criterion lazy 8 performance as suggested by 
Air Training Command flight manuals and as substantiated empirically in previous studies (Knoop & Welde, 
1973)), This is only one example of hundreds where simultaneity of significant events bears on successful 
performance, and the role of the Relative computational mechanism is to explore the relevance to 
performance measurement of conditional probabilities. When one considers the plausible theory that much 
of performance on continuous control tasks can be modeled by discrete successive acquisitions of key 
states, the value of the Relative type of measure and its role in the processor becomes clear. 

The Absolute computational mechanism essentially amounts to computing a discrete version 
(Hadainard Transform) of the Fourier Transform, wherein the power of various frequency components of 
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the signal (BTS) is assessed. Each transform operation produces a number of coefficients (correlation 
values), which are summarized by their mean and variance. Previous work (Connelly, et. al., 1969) has 
shown that the variance of the Hadamard coefficients is most useful in discriminating perfomiances of 
various skill levels. The Absolute mechanism generates these variances for subsequent validation testing, 
described next. 

Validation Tests 

a. Description. For the vast majority of performance tasks, there is no single necessary and sufficient 
test that can be applied to candidate measures to assess their validity. Measures which appear to have 
content validity often fail to reliably discriminate even between novice and highly experienced performers. 
Measures which appear to have concurrent validity may or may not satisfy other validation criteria, 
depending on the reliability and sensitivity of the metric used as a basis of comparison. 

The approach in this study was to develop three empirically-based validation tests to be appHed by 
the measurement processor. Collectively, the tests are used to determine the likelihood that each cai^didate 
measure is valid. Final analysis and assurance of the measure's content validity is perfonned by the user of 
the processor, based on the evidence accrued by it and printed out for his consideration. 

The first test assesses the measure's potential contribution to discriminating between performances at 
opposite ends of the skill continuum. The data employed for this test are selected by the user. For the 
T-37 pilot performance tasks that were to have been addressed here, the following two types of data would 
have been investigated: 

(1) Flights flown by instructor pilots to demonstrate their best performances and simulated novice 
performances of each maneuver. 

(2) Flights flown by students at the neophyte stage and at the successful completion of training. 

The techniques implemented to apply this first test include: (a) comparison of residues from regression 
analyses, and (b) the rank sum statistic (see Validation Tests section). 

The second test assesses the measure's functional relationships with variables such as number of trials 
and time in training. A measure which demonstrates that learning has occurred from neophyte to 
experienced levels of performance would posses a higher likelihood of validity than one which consistently 
does not, for example. Again, the data to be employed for this test are specifiable by the user. For the T-37 
pilot tasks, the following data would have been experimented with: (1) time in training, (2) Number of 
practice sorties on the maneuver, and (3) number of practice trials on the maneuver. The technique used to 
apply this test consists of developing and analyzing a multi-variable regression function. (An alternative 
technique based on the use of Markov learning models was conceived, but due to lack of data, has not yet 
been developed to the point of implementation.) 

The third test assesses the measure's functional relationships with subjectively derived ordinal scale 
measures of performance. Measures which tend to reinforce the subjective ordering of performances are 
considered more likely to be valid than those which consistently fail to do so. The data employed for this 
test, as with the other tests, are specified by the user. For the T-37 tasks, instructor pilot ratings would have 
been investigated for use. The technique for applying the test is to develop and analyze multi-variable 
regression functions, as in the second test described in the preceding paragraph. 

The regression techniques used for applying some of the above validation tests were described 
previously. The rank sum test is -described next. 

b. Rank Sum Statistic. The cornputer-aided generation of performance measures requires the 
systematic generation and evaluation of many candidate measures. It is necessary to assess these measures' 
potential contribution to overall performance measurement. One aid in accomplishing this using the rank 
sum statistic was developed for investigation in this study. 

Consider a process where data are available from two performance classes (e.g., flights produced by 
instructor pilots and flights produced by neophyte student pilots). Candidate proficiency measures will 
a yield two sets of quantitative variable values when applied to the data from these two performance classes. 
It is possible to test these sets to determine if they come from dilTerent parent distributions. If they do 
come from different parent distributions and there is little overlap in the distribution functions, then the 
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candidate measures are highly likely to be useful in proficiency assessment; i.e., there is a high probability 
that the tested measures will satisfy other validation 'criteria. On the other hand, if these two distributions 
have considerable overlap, then the measures would probably not be very useful. 

A test for determining the degree of similarity of the two parent distributions can be selected from 
statistical analysis hypothesis testing where one can initially assert that the two distributions are the same 
(null hypothesis). One method of testing "the hypothesis that distributions F, (X) and F2(X) are equal is the 
rank sum test. This is a simple non-parametric test which indicates the likelihood that two sets of data, 
which may be of different sizes, come from the same distribution. 

Tej|t^of the hypothesis that Fi(X) is equal to F2(X) is developed as follows: 

Let X, , X2, . . . , Xp J , and Yi , Y2, . . . , Y^^ 

denote random samples of two sizes, nj and n2 , taken from populations with continuous density functions 
F,(X) and F2(X), respectively. Let these two sets of samples be ordered in increasing magnitude and 
combined to a single ordered set where a possible arrangement miglit be as follows: 

Yi , Y2 , Xi , Y3 , X2 , Y4 , etc. 

Of special interest is the sum of the ranks of the smaller set nj (where ni ^ n2). (For example, the ranks of 
the X's are 3, 5, etc.) The sum of these ranks is a statistic of known distribution for given values of nj and 
n2. Therefore, the statistic value can be used as an indicator that the hypothesis Fi(X) = F2(X) is valid. 
Table 13 gives the critical values or limits of a 95 percent confidence interval for small values of ny and h2 . 
(The significance level of the rank sum test is not preserved if the two populations differ in dispersion or 
shape. Whether or not they differ in this way is expected to depend on the measure under test. Plans to 
analyze this empirically for the various measures on the T-37 problem and, as required, develop techniques 
to account for observed effects did not materialize due to inability to collect required data,) This table is 
taken from (Hoel, 1962) and applies for values of nj and n2 less than 10. For larger sample sizes the 
distribution is approximated closely by the normal distribution with a mean and variance given as follows: 

Mean = ni (n, + n2 + l)/2 

ni n2 (ni + n2 + I) 
Variance = 

12 



V. SUMMARY AND CONCLUDING REMARKS 

A computer-aided system has been developed and implemented for use in deriving and validating 
measures of operator performance. Its uniqueness is characterized by: (I) a logical division of human and 
computer-processor functions, integrated through an interactive man/machine systems approach to 
measurement research; (2) an experimental approach to deriving measures by generating vectors which span 
various conceivable measure spaces and operating on the vectors using multiple regression analysis; and (3) a 
systematic empirical approach to validation-testing of candidate measures to assess their likelihood of 
contributing to overall performance measurement. 

One of the most important features desired in the processor is its ability to automatically generate 
and test candidate performance measures with a miniumum of inputs from the user. The processor 
successfully implements this desired feature in that it reads in raw performance data and prints out tested 
performance measures. To do this, it first automatically performs data smoothing; i.e., removal of noise in 
the data. It then performs logical sectoring in which maneuvers are automatically divided into sectors that 
can be conveniently analyzed in subsequent processing. Next, the processor automatically applies a 
regression analysis procedure to establish criterion performance in the form of simple regression functions. 
The independent variables for each function were selected based on their being monotonic over the 
maneuver sector of interest. Finally, the processor applies adaptive mathematical models to the data, and 
based on types of deviations (BTS) from the criterion performance functions, generates and tests for 
validity a variety of performance measures. It is truly an automatic processor. 
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Table 13. Rank Sum Critical Values* 
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* This tabic was c.vtrnctcd from a more complete tabic (A-20) in Intloa'uctioa to 
Sfnfi'sficaL iiKi/ysis, ?.nd edition, by \V. J. Oixon and P. J. Massey^ with pornlission from 
Uic publishers, the McGraw-Uiil Book Company. } 
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The types of measures generated and tested were defined a priori in terms of underlying 
characteristics which render them suitable candidates. One type (Relative) represents probability measures 
related to significant event proximities. Another (Absolute) represents measures of system variable 
frequencies and periodicities. A^third (State Transfer) represents measures of state transitions that occur 
over time, including system divergence from or convergence upon criterion terminal performance. A fourth 
type (State Frequency) that was identified but not specifically addressed in this effort uses data generated 
by the State Transfer computational mechanism to address measures of reference function deviations. 

To investigate the above measures, vectors were identified which constitute generators of the measure 
spaces corresponding to. each measure-type. The measures thereby spanned are explored by the various 
computational mechanisms using regression analysis and a number of empirical validation tests. Table 14 
summarizes the measure spaces, the components of vectors by which they are spanned, and the basic 
functions performed by each computational mechanism. 



Table 14, Summary of Measure Space^> 
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* Separate computational mechanism not yet implemented in processor. 



The success of the automatic maneuver sectoring is a main factor in processor effectiveness. It allows 
use of simple regression functions for describing criterion performance since small portions of the 
maneuvers can be treated separately. Had this automatic sectoring not been feasible, then a considerably 
more complicated regression function would have been required; i.e., it would have been necessary to 
attempt to model the entire maneuver or large portions thereof with a single regression function. 
Preliminary evaluation of the automatic sectoring using two excellent and two poor maneuvers, as rated and 
flown by IP's, indicates that the sectoring will work in a satisfactory way over a range of maneuver 



demonstrations. Flowever, since the beginning of each sector is detected when a specific variable amplitude 
exceeds a threshold limit, it is possible that some maneuver demonstrations (especially those produced by 
neophyte students) presented to the processor will not be processed properly. Thus, before the evaluation 
of tjae automatic sectoring can be considered complete, distributions of these key variable values are 
required over the range of expected perfomiance demonstrations; i,e., student demonstrations from 
neophyte to skilled. 

Another key area in the development of the processor is the use of a simple regression function that 
uses the "five sums approach" (Connelly et al, 1969, pp. 179-181), so that additional infomiation can be 
simply added to the processor as it becomes available. Initial evaluation using two excellent and two poor 
demonstrations of each maneuver type indicates that the mechanism for developing the satisfactory 
criterion function is available. It should be noted, however, that although the mechanisms for producing the 
criterion functions exis't in the processor, the data itself must be studied using additional demonstrations of 
flight performance in order to determine if excellent perfonnance ^data is clustered about the criterion 
functions. Such clustering is necessary for establishing useful criterion functions. Should clustering of 
excellent performance data fail to materialize, as evidenced by a large residual value, maneuver parameters 
such as 0Max and 0Max may have to be included in the regression functions. 

The adaptive mathematical models developed and experimented with in earlier studies have now been 
refined and adapted to use in an automatic processor. Future refinements beyond those now implemented 
may be easily invoked as required vdue to the modular design -of the software. Whether or not further 
refinements are necessary or desirable could not be determined in this effort due to the previously 
mentioned unavailability of sample performance data. However, the central features of what is believ.ed to 
be powerful and highly useful measurement research tool have been successfully implemented; and 
hopefully the underlying theoretical concepts and the implementation techniques that were developed and 
documented herein will, as a minimum, serve to inspire further measurement work along these lines. . 
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13.83 


91.07 


e 


0.64 


0.48 


.98 


1 .17 


19.01 


.... 


0.62 


0.48 


.93 


.68 


7.07 


^/samples 




11 
7.0 
6.80 


5 

7.3 
4.10 


13 
6.9 
3.3 


8 

10.8 
-3.9 


FUNCTIONAL FORMrDV = + 




3 M 4 M 






REFERENCE FUNCTIONS 




Variable 


B ■ 
o 






^3 




h 


250.10 


-120.60 


150.80 


-21.94 


' -13.55 




.85 


.99 


1.61 


-.09 


-.01 


AS 


398.00 


5.18 


-7,50 


-25.05 


* 2 o S 2 


d 


-24.03 


. 5.44 


3.12 


3.82 


-.81 




-5.38 


0.55 


5.80 


0.68 


.00 



66 

70 



mani:uv[;r n'Pc i.azy 8 



SECTOR 2 



ERFlOIvS 





r;-:coll':iit 


Poor 


Dependent 
Variable 


<_ 

C 


Good Plus 
102 


108 


113(L) 


106(R) 


h 

X X 


53.88 


64.95 


53.06 


483.84 


820.75 


T 


0.09 


0.81 


.96 


9.32 


11.76 


AS 


1.42 


i.21 


i.5S 


27.81 


37.23 


6 


0.31 


0.27 


0.34J 


6.44 


3.20 






















#/samples 




19=^ 

27.6 

44.4 


25 

31.4 
45.8 


19 

26.6 
40.6 


25 

43.1 

56.5 



FUNCTIOR^L FORM: DV = + B^«^ + B^<t> + B^^j^ + 8^0^ 

REFERENCE FUNCTIONS 



Variable 


B 

o 


^1 




^3 




h 


-4410.00 


32.47 


-0.02 


53.69 


58.31 




-7.80 


0.46 


0.00 


.02 


0.08 


AS 


387.7 


-1.18 


-0.00 


0.26 


-4.08 


0 


-53.07 


1.27 


-O.OI 


.75 


.69 


0 













Q 67 

ERIC ' ,71 



MANHUVCR TYPP. T-^z7 0 SECTOR 3 

ERRORS 





Cxcollcnt 


Poor 


Dependent 
Variable 


e 

C 


Good Plus 
102 


1 o r> fx \ 

128(L) 


1 1 3 (L) 


106(R) 


■ 

h 


14.83 


17.65 


11 .78 


526.14 


3010.40 




3.30 


3.93 


2.63 


37.07 


42.26 


MO 


1.63 


1.94 


1.29 


32.04 


173.65 


e 


1.80 


2.03 


1 .54 


12.41 


84.63 
























#/saniples 




14 

27.4 
73.9 


13 

31.3 
89.1 


12 

26.2 
76.7 


13 

40.9 
84.9 


FUNCTIONAL FORM: DV = 




^ + B-^-. + B.0.- 
3 M 4 M 






REFERENCE FUNCTIONS 




Variable 


B 

o 




\ 






h 

♦ 


-4848.00 


38.43 


-0.18 


328.9 


-61.06 




-3.68 


2.47 


-.00 


3.03 


-1.81 


AS 


{402.00 

i" 


-2.94 


0.01 


-17.70 


4.49 


e 


-1-4.20 


0.91 


-0.01 


8.50 


-1.61 















68 

no 



MANEUVER TYPC lazy 8 



SECTOR -1 



ERRORS 





Excellent 


Poor 


Dependent 
Variable 


C 


Good Plus 
102 




1 loU.j 


1 Uo U<J 


h 


5* 

29.65 


28.18 


31.46 


325.76 


1912.90 


a 


3.65 


■ 2.83 


4.43 


33.30 


318.12 

- 


A C 


2.14 


2.12 


2.17 


3.03 


602.64 


6 














2.82 


2.47 


3.24 


7.96 


249.63 












4f/sa moles 




16 
-27.2 
■ 75.9 


13 
-25.0 
90.0 


14 
-26.8 
82.6 


17 
-50.7 
84.7 


FUNCTIONAL FORM: DV = 






Vm. 






REFERENCE FUNCTIONS 




Variable 


B 

o 


^1 








h 


390.50 


-8.05 


-0.51 


32.63 


26.75 




-394.10 


.78 


0.03 


-13.25 


1 . 65 


AS 


-732.2 


0.48 


0.06 


-21.96 


3.31 


$ 














331 .80 


■ -57 


-0.05 


7.52 


-.64 



ERIC 



69 

73 



MANEUVCR TYPL: Lazy 0 



SECTOR 5 



ERRORS 







Poor 1 


Dependent 
Variable 


C 


Cvoocl Plus 
102 


lOO(L) • 


113{L) 


106(R) 


h 


24.66 


32.17 


19.02 


563.96 


2892.20 




0.35 


.40 


„31 


17.80 


22.28 




1.65 


2.11 


1.30 


22.98 


60.16 




0.96 


1 . oU 






7 ?fi 














^f/samples 


1 1 


18 
-27.1 

48.6 ■ 


24 
-24.7 
50.6 


21 
-26.1 
50.6 


19 
-47.6 
44.7 


FUNCTIONAL FORM: DV = 










REFERENCE FUNCTIONS 




Variable 


B 

o 










h 


-2083.00 


13.86 


0.14 


151.-60 


• 126.00 




159.70 


0.08 


-0.01 


-2.07 


5o83 


AS 


-35.85 • 


0.73 


-0.00 


-3.99 


2.88 


0 


-14.89 


0.68 


0.00 


0.16 


0.34 


0 













70 



'74 



MANEUVER lYPE Normal Landing ' SECTOR 1 



i;rrors 





rxccJlcnt 


Poor 


Dependent 
Variable 


c 


1 Ct 


191 


1 9"^ 




n 


57.17 


39.85 


78.58 


120.03 


100.33 


Y 


0.75 


.70 


.80 


50;68 


18.20 


AS 


2.52 


2.41 


2.66 


49.49 


29.34 


e 


0.65 


0.77 


„51 


1 .02 


<,83 




1 .40 


1 .22 


1 .62 


1.48 


2.24 










••|r/samples 




42 

-2.2 


34 

-2.9 


33 

-3.2 


39 

-2.4 






7.3 


-12.7 


-6.0 


7.7 


FUNCTIONAL FORM 


: DV = 


+ Bjt + b/ 


' ^ h% * 


«4^M . 






REFERENCE FUNCTIONS 




Variable 


B 

o 






i 

^3 




h 


-128.20 


351.40 


-452.60 


12.65 


1 . 64 




-5.47 


-12.69 


9.28 


-92.43 


3.10 


AS 


5.24 


-16.54 


11.02 


-82.86 


2.90 


$ 


-1.28 


-2.85 


2.22 


-0.27 


0.05 




-3.85 


'. 8.62 


-1.37 


.56 


.13 



ERIC 75 



N4ANEUVCR TYPE Nonnal I.anclincj 



SfXTOR 2 



ERRORS 





Excollonl 


Poor 


1 1 9^ /^i ♦•X ^ 

iJcpcnacni 
Variable 


C 




121 


1 23 


124 


h 
* • 


6.57 


8,56 


4,50 


390.63 


1687.60 


T 


7.92 


8.51 


7,30 


... 28.28 


252.81 


AS 


0.98 


.99 


.97 


39.20 


306.01 


d 


0,82 


.91 


.73 


1.47 


1.84 


4> 


5.46 


7.08 


5.80 


11.09 


104.93 


#/samples 




24 


23 


18 


29 






2.4 - 


1.6 


3.5 


-4.7 




64.5 


57.6 


70.4 


57.8 


FUNCTIONAL FORM: DV = 








REFERENCE FUNCTIONS 




Variable 


B 

o 






h 




h 


-584.50 


154.20 


-88.40 


256.60 


-4.39 




310.20 


147.20 


34.72 


30.41 


-2.45 


AS 


313.90 


-56.10 


10.35 


47.17 


-3.29 


6 


- . 00 


2.84 


-3.53 


-0.05 


0.01 


4> 


-59.34 


142.6 


-125.00 


-15.52 


1.85 



ERIC 



72 

•7G 



M/\NF;UVI:R type Normal Landing 



SECTOR 3 



LURORS 





Excolk nt 


Poor 


Dependent 
Variable 


C 


122 


121 


123 


124 


h 


32.64 


28.67 


37.71 


943.50 


503.05 




0.83 


0.73 


0.95 


1606.30 


465.32 


As 


1 .80 


1.76 


1 .86 


1365.00 


387.16 


e 


0.82 


.61 


1.10 


13.03 


1 . 09 




1.77 


1.77 


1 .85 


71.38 


24.31 


Ti/samplcs 




23 

-1 .91 
8.07 


18 

4.44 
17.90 


25 

5.26 
28.6 


37 

-3.93 
7.7 


FUNCTIOl^AL FORM: DV = 


4 

+ B t + Bjt 










REFERENCE FUNCTIONS 




Variable 


B 

o 


h 


^2 






h 


-1445.00 


22.47 


-35.08 


-169.40 


108.50 




2278.00 


-6.14 


3.62 


260.60 


-169.30 


AS 


1758.00 


-28.66 


-3.45 


221.60 


-144.40 


.J 


-13.81 


0.85 


-0.03 


-1.31 


1.25 


, ■ 


92„24 


-35.60 


32.43 


12.44 


-7.75 



ERIC 



73 

■i77 



^4AN^;UVl:H 'HVE Nornu-il bonding 

IlRROr^S 



sr;cTOR 4 





r,:-:collont 


Poor 


Dcpsndeiit 
.Variable 


€ 

c 


122 


121 


123 


124 


h 


12.03 


14,36 


9.70 


912.02 


593.75 




3.08 


3.02 


3.15 


39.75 


41.51 


AS 


3.21 


3.74 


2.67 


28.68 


24.70 


d 


0.98 


1.08 


.89 


1.60 


2.00 


<> 


4.12 


4.42 


3 . 83 


13 97 


9 58 












^/samples 




40 

-6.69 . 
32.5 


40 

-5.96 
32.3 


42 
-10.6 

37.5 


47 

-6.14 
27.0 


FUNCTIONAL FORM: DV = Bq 


-f B^t 4- B^t 










REFERENCE FUNCTIONS 




Variable 


B 

o 


h 








h 


2759.00 


-639.10 


38.91 


181.90 


-^^sf^ OA 




165.90 


130.40 


32.90 


-2.53 


7.11 


AS 


263.00 


-40.75 


31.79 


1.17 


-3.86 


e 


12.59 


-12.33 


^ 10.93 


0.56 


-0.32 




56.93 


. 13.35 


-9.62 


1.99 


-0.75 



74 

78 



K1ANI:UV1.;R type Nonnal Landing 



SECTOR S 



Dependent 
Vori<\blc 



122 



121 



Poor 



123 



124 



h 

AS 

e 



4.65 
1.25 
2.85 



5.77 
1 .42 
3.47 
0.61 
1 .72 



3.26 
1.04 
2.09 



1032.30 
106.64 
309.46 
6.66 
17.95 



1009.30 
137.91 
411,38 
10.61 

24 . 6a 



0 



'M 



#/samples| 31 25 25 32 

Or., 1 G.G 4.9 -5.1 -6.G 

6.9 -8.4 -7.3 9.7 

1 — ^_ 

FUNCTIONAL FORM: DV = + B^t + B^t^ + + ^a\\ 

REFERENCE FUNCTIONS 



Variable 


8 

o 






B 

3 


^4 


h 


-507.90 


-319.10 


314.40 


-51.31 


-4.59 




520,70 


0.51 


5.04 


10.34 


-1.17 


AS 


-62.60 


5.44 


-26.42 


30.57 


-3.38 


e 


0.84 


• 1.53 


5.99 


-0.58 


0.07 




-G.62 


:-22.60 


10.85 


1.87 


-0.14 



75 

'79 



MANf.UVER TYVL Morrnnl Landing 

ERRORS 



SECTOR 6 





Excel k'nt 


Poor 


Dopcndcnt 
Variable 




122 


121 


123 


124 


h 




8.59 




854.85 


18160.00 










216.71 


12365.00 


AS 




1.44 




2.10 


249.33 


e 




.22 




0 .53 


27.51 






.13 




0.12 


2.88 


^/samples 




12 

5.5 
-0.5 


8 

3.6 
-0.3 


8 

3.8 
-0.3 


14 
8.9 

-2,4 


S 
: 

FUNCTIONAL FORM: DV = 


+ B^t + b/ + B^e^ -f 








REFERENCE FUNCTIONS 




Variable 


B 

o 




^2 




^4 


h 


-96.22 


141.10 


-19.52 


1117.00 


1 1920.00 




266,40 


-6.45 


7,08 


-782,4 


-7QQ0 no 


AS 


90.57 


-11.15 


7.98 


-16.94 


-159.00 


e 


-1.04 


0.66 


-0.05 


-0.37 


-15.69 




0.04 


, -1 .02 


0.46 


0,34 


2.77 



76 

80 



M/\NEUVrR TYPE narrcl Roll 

i:khop>s 



SECTOR 1 





n/:col!c-nt 


F'oor 


Dependent 
Variable 


'c 


101 


107 


iHa) 


IIO(L) 


h 


0.20 


.25 


0.10 


7126. 60 


62614.00 














AS 


0.04 


0.00 


0.04 


23.60 


120.92 


d 


U , U 


n nn 

u • UU 




2 95 


7. 78 




0.03 


0.00 


0.04 


.75 


4.24 


Tr/sarnples 




1 

-9.8 


4 

-9.5 


8 

-9.1 


32 

-9.5 




1.4 


2.4 


5.2 


-10.3 


FUNCTIONAL FORM: DV = B + B t + B 

0 1 2 










REFERENCE FUNCTIONS 




Variable 


• B 

o 








\ 


h 


184.40 


-89.60 


16.00 


-3,53 


-81.04 














AS 


275.00 


-2.09 


4.39 


3.48 


-11.17 


$ 


5.28 


-2.26 


-.40 


1.25 


-0.06 




2.30 


■ -2.32 


1 .60 


3.12 


0.74 



ERIC 



77 

'81 



MAN r.UVr.R TYPE Bon-ol Koll SECTOR_2 



KRROUS 





!;:-:c'--Ucnl 


Poor 


Dependent 
Variable 


c 


101 


107 


1H(L) 


IIO(L) 


n 


4G.60 


37.67 


54.45 


198.03 


2.98.18 




1 .77 


2.37 


1 .24 


• 8.26 


8.55 


AS 


2.62 


2.31 


2.89 


34.87 


17.56 


e 


0.81 


0.89 


.74 


2.67 


6,76 




4.79 


4.94 


4.65 


11.77 


5,51 


4r/ samples 




29 
-13.1 
-44.9 


33 
-14.7 
-37.7 


32 
-18.7 
-37.5 


26 
-10.1 
-41.8 

.. 1 


FUNCT] 


[ONAL FORM: DV = B^^ ■ 


f B t + B 

1 M 




B 0 
4 M 






REFERENCE' FUNCTIONS 




Variable 


B 

o 


^1 






1 


n 


-1559,00 


-1987.00 


758.90 


-21.96 


-32.10 


\L 
r 


5.75 


5o73 


• -35.52 


-0.00 


0.12 

■ 


MO 


60.18 


87.77 


-46.18 


-6.51 


-1.62 


e 


-8.89 


-41 .90 


56.58 


0.09 


0.07 




-12.73 


-214.70 


202 . 10 


-2.66 


0.05 



78 



8.2 

, ' > 



MANEUVER lYPE Bcirrc-i Roll SECTOR 3 



ERRORS 





Excollont 


Poor 


Dependent 
Variable 


e 

c 


101 


107 


114(9) 




h 


15.54 


16.66 


14.49 


620.28 


229.42 




3,03 


3.22 ' 


2.85 . 


• 7.14 


20.24 


AS 


2.08 






/v . /y 


13.95 


e 


0.65 


.71 


.60 


2.48 


•' 4.70 
























#/samplo5 




14 • 

34.2 

83.9 


15 

38.1 

85.0 


15 

38.4 

81.3 


17 

26.4 

84,5 


FUNCTIONAL FORM 


* 

: DV ~ B + 
0 


^1^ -^^2^' h^M^^ 


4 M 






REFERENCE FUNCTIONS 




Variable 


B 

o 






^3 


B 
^4 


h 


7751.00 


22.03 


-.07 


-23.32 


-95.66 


4' 


117.40 


0.10 


0.00 


-1.51 


-1.03 


AS 


1579.00 


-1.02 


0.00 


3.90 


-17.28 


0 


14.89 


0.95 


-0.00 


0.94 


-.51 


o 













ERJC 83 



^4^N^,UV^,R TY.Pi: Barrel Roll 



SECTOR 4 



rRROK.S 











Poor 


Dependent 
Variable 


t 

c 


101 


107 






h 


6i02 


6.05' 


5.99 


250.00 


1644.00 


T 


3.32 


3.36 


3.27 


80.01 




AS 


0.69 


.77 


.61 


1.33 




e 


n 9? 


n 1 7 




7. 28 
















Tf/samples 




9 


9 


11 


1 0 






31 .2 


34.6 


34.9 


-24.5 




169 .0 


170.0 


172.0 


168.0 


FUNCTIONAL FORM: DV = 


+ Bj0 + B^<t) 










REFERENCE FUNCTIONS 




Variable 


B 

o 










h 


-169.4 


20^30 


-.05 


-31.13 


0.34 




848.5 


2.18 


-0.00 


-13.08 


-3.25 


AS 


801.50 


-1.33 


0.00 


-2.87 


-2.53 


0 


-147.00 


.01 


0.00 


0.83 


0.95 


<> 













80 



MANCUVIIR TYPi: i?.arrol Moll 



SECTOR 5 



r.RRORS 





r;-:c^'ll-"nl 


Poor 


Dependent 
Vafia!:)lc 


i 

c 


101 


107(L) 


1 1 4 (1.) 




h 


3.40 


3.79 


3.01 




251.04 






3.58 


3.50 


3.66 




ion AC 

1 29 . 45 




AS 


0.77 


.88 


0.65 




7.90 




0 


0.85 


0.44 


1.26 




15.19 








1 










If/ Scimpics 




9 

-35.1 
268.0 


9 

-35.2 
263.0 


7 

-41.3 
260.0 




FUNCT] 


■QNAL FORM: DV = B^^ 


+ B^ 0 Tb^^'^ 










REFERENCE FUNCTIONS 






Variable 


B 

o 








^^4 


h 


-3178.00 


18.06 


-0.05 




18.64 


10.96 




-2158.00 


2.83 


-0.00 




6.50 


8.28 


AS 


-509.90 


-1 .01 


0.05 




-0.57 


2.80 




73.60 


0.87 


0.00 




-0.89 


0.05 

















81 

ER?C • 8fi 



MANEUVKIl TYPE Bnrrcl Roll 



SECTOR 6 



ERRORS 





Excellent 




Dependent 
Variable 


C 


i U i 


in? 






h 


21 .48 


26.47 


16.48 


100.83 




T 


3.07 


3.32 


2.83 


192.91 




AS 


1.10 


1.51 


.70 


210.67 




e 


0.69 


.46 


.42 


135.99 
















Tt/samples 




11 
-36.5 

349,0 


11 
-37,6 
353.0 


6 

-44.2 
348.0 




FUNCTIONAL FORM 


: DV = Bq - 


I- B <^ + B 0 




^4^M . 






REFERENCE FUNCTIONS 




Variable 


B 

o 






h 


^4 


h 


-576.90 


-5 .60 


-0.01 


14.63 


10.55 




-311.70 


-4.22 


0.00 


21.86 


5.28 


AS 


-118.50 


0.58 


0,00 


29.96 


4.18 


0 


-289.90 


-4.36 


0.00 


18.90 


4.53 















82 

ErJc 8G 



